Deep learning techniques for magnetic resonance image reconstruction

ABSTRACT

A magnetic resonance imaging (MRI) system, comprising: a magnetics system comprising: a B 0  magnet configured to provide a B 0  field for the MRI system; gradient coils configured to provide gradient fields for the MRI system; and at least one RF coil configured to detect magnetic resonance (MR) signals; and a controller configured to: control the magnetics system to acquire MR spatial frequency data using non-Cartesian sampling; and generate an MR image from the acquired MR spatial frequency data using a neural network model comprising one or more neural network blocks including a first neural network block, wherein the first neural network block is configured to perform data consistency processing using a non-uniform Fourier transformation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Application Ser. No. 62/711,895, Attorney Docket No.00354.70028US00, filed Jul. 30, 2018, and titled “DEEP LEARNINGTECHNIQUES FOR MAGNETIC RESONANCE IMAGE RECONSTRUCTION”, U.S.Provisional Application Ser. No. 62/737,524, Attorney Docket No.00354.70028US01, filed Sep. 27, 2018, and titled “DEEP LEARNINGTECHNIQUES FOR MAGNETIC RESONANCE IMAGE RECONSTRUCTION”, U.S.Provisional Application Ser. No. 62/744,529, Attorney Docket No.00354.70028US02, filed Oct. 11, 2018, and titled “DEEP LEARNINGTECHNIQUES FOR MAGNETIC RESONANCE IMAGE RECONSTRUCTION”, and U.S.Provisional Application Ser. No. 62/820,119, Attorney Docket No.“00354.70039U500”, filed Mar. 18, 2019, and titled “END-TO-END LEARNABLEMR IMAGE RECONSTRUCTION”, each of which is incorporated by reference inits entirety.

BACKGROUND

Magnetic resonance imaging (MRI) provides an important imaging modalityfor numerous applications and is widely utilized in clinical andresearch settings to produce images of the inside of the human body. MRIis based on detecting magnetic resonance (MR) signals, which areelectromagnetic waves emitted by atoms in response to state changesresulting from applied electromagnetic fields. For example, nuclearmagnetic resonance (NMR) techniques involve detecting MR signals emittedfrom the nuclei of excited atoms upon the re-alignment or relaxation ofthe nuclear spin of atoms in an object being imaged (e.g., atoms in thetissue of the human body). Detected MR signals may be processed toproduce images, which in the context of medical applications, allows forthe investigation of internal structures and/or biological processeswithin the body for diagnostic, therapeutic and/or research purposes.

MRI provides an attractive imaging modality for biological imaging dueto its ability to produce non-invasive images having relatively highresolution and contrast without the safety concerns of other modalities(e.g., without needing to expose the subject to ionizing radiation, suchas x-rays, or introducing radioactive material into the body).Additionally, MRI is particularly well suited to provide soft tissuecontrast, which can be exploited to image subject matter that otherimaging modalities are incapable of satisfactorily imaging. Moreover, MRtechniques are capable of capturing information about structures and/orbiological processes that other modalities are incapable of acquiring.However, there are a number of drawbacks to conventional MRI techniquesthat, for a given imaging application, may include the relatively highcost of the equipment, limited availability (e.g., difficulty andexpense in gaining access to clinical MRI scanners), and the length ofthe image acquisition process.

To increase imaging quality, the trend in clinical and research MRI hasbeen to increase the field strength of MRI scanners to improve one ormore specifications of scan time, image resolution, and image contrast,which in turn drives up costs of MRI imaging. The vast majority ofinstalled MRI scanners operate using at least at 1.5 or 3 tesla (T),which refers to the field strength of the main magnetic field B0 of thescanner. A rough cost estimate for a clinical MRI scanner is on theorder of one million dollars per tesla, which does not even factor inthe substantial operation, service, and maintenance costs involved inoperating such MRI scanners. Additionally, conventional high-field MRIsystems typically require large superconducting magnets and associatedelectronics to generate a strong uniform static magnetic field (B0) inwhich a subject (e.g., a patient) is imaged. Superconducting magnetsfurther require cryogenic equipment to keep the conductors in asuperconducting state. The size of such systems is considerable with atypical MRI installment including multiple rooms for the magneticcomponents, electronics, thermal management system, and control consoleareas, including a specially shielded room to isolate the magneticcomponents of the MRI system. The size and expense of MRI systemsgenerally limits their usage to facilities, such as hospitals andacademic research centers, which have sufficient space and resources topurchase and maintain them. The high cost and substantial spacerequirements of high-field MRI systems results in limited availabilityof MRI scanners. As such, there are frequently clinical situations inwhich an MRI scan would be beneficial, but is impractical or impossibledue to the above-described limitations and as described in furtherdetail below.

SUMMARY

Some embodiments are directed to a method comprising: generating amagnetic resonance (MR) image from input MR spatial frequency data usinga neural network model that comprises: a first neural network sub-modelconfigured to process spatial frequency domain data; and a second neuralnetwork sub-model configured to process image domain data.

Some embodiments are directly to a system, comprising at least onecomputer hardware processor; and at least one non-transitorycomputer-readable storage medium storing processor-executableinstructions that, when executed by the at least one computer hardwareprocessor, cause the at least one computer hardware processor toperform: generating a magnetic resonance (MR) image from MR spatialfrequency data using a neural network model. The neural network includesthat comprises: a first neural network portion configured to processdata in a spatial frequency domain; and a second neural network portionconfigured to process data in an image domain.

Some embodiments are directed to at least one non-transitorycomputer-readable storage medium storing processor-executableinstructions that, when executed by at least one computer hardwareprocessor, cause the at least one computer hardware processor toperform: generating a magnetic resonance (MR) image from MR spatialfrequency data using a neural network model. The neural network modelcomprises a first neural network portion configured to process data in aspatial frequency domain; and a second neural network portion configuredto process data in an image domain.

Some embodiments are directed to a method, comprising: generating amagnetic resonance (MR) image from input MR spatial frequency data usinga neural network model that comprises a neural network sub-modelconfigured to process spatial frequency domain data and having a locallyconnected neural network layer.

Some embodiments are directed to a system comprising: at least oneprocessor; at least one non-transitory computer-readable storage mediumstoring processor-executable instructions that, when executed, cause theat least one processor to perform: generating a magnetic resonance (MR)image from input MR spatial frequency data using a neural network modelthat comprises a neural network sub-model configured to process spatialfrequency domain data and having a locally connected neural networklayer.

At least one non-transitory computer-readable storage medium storingprocessor-executable instructions that, when executed, cause the atleast one processor to perform: generating a magnetic resonance (MR)image from input MR spatial frequency data using a neural network modelthat comprises a neural network sub-model configured to process spatialfrequency domain data and having a locally connected neural networklayer.

Some embodiments provide for at least one non-transitorycomputer-readable storage medium storing processor-executableinstructions that, when executed by at least one computer hardwareprocessor, cause the at least one computer hardware processor to performa method comprising: generating a magnetic resonance (MR) image frominput MR spatial frequency data using a neural network model comprisingone or more neural network blocks including a first neural networkblock, wherein the first neural network block is configured to performdata consistency processing using a non-uniform Fourier transformationfor transforming image domain data to spatial frequency domain data.

Some embodiments provide for a magnetic resonance imaging (MRI) system,comprising: a magnetics system comprising: a B₀ magnet configured toprovide a B₀ field for the MRI system; gradient coils configured toprovide gradient fields for the MRI system; and at least one RF coilconfigured to detect magnetic resonance (MR) signals; a controllerconfigured to: control the magnetics system to acquire MR spatialfrequency data; generate an MR image from MR spatial frequency datausing a neural network model that comprises: a first neural networkportion configured to process data in a spatial frequency domain; and asecond neural network portion configured to process data in an imagedomain.

Some embodiments a magnetic resonance imaging (MRI) system, comprising:a magnetics system comprising: a B₀ magnet configured to provide a B₀field for the MRI system; gradient coils configured to provide gradientfields for the MRI system; and at least one RF coil configured to detectmagnetic resonance (MR) signals; a controller configured to: control themagnetics system to acquire MR spatial frequency data; generate an MRimage from input MR spatial frequency data using a neural network modelthat comprises a neural network sub-model configured to process spatialfrequency domain data and having a locally connected neural networklayer.

Some embodiments provide for a method, comprising: generating a magneticresonance (MR) image from input MR spatial frequency data using a neuralnetwork model comprising one or more neural network blocks including afirst neural network block, wherein the first neural network block isconfigured to perform data consistency processing using a non-uniformFourier transformation for transforming image domain data to spatialfrequency domain data.

Some embodiments provide for a system, comprising: at least one computerhardware processor; and at least one non-transitory computer-readablestorage medium storing processor-executable instructions that, whenexecuted by the at least one computer hardware processor, cause the atleast one computer hardware processor to perform a method comprising:generating a magnetic resonance (MR) image from input MR spatialfrequency data using a neural network model comprising one or moreneural network blocks including a first neural network block, whereinthe first neural network block is configured to perform data consistencyprocessing using a non-uniform Fourier transformation for transformingimage domain data to spatial frequency domain data.

Some embodiments provide for a magnetic resonance imaging (MRI) system,comprising: a magnetics system comprising: a B₀ magnet configured toprovide a B₀ field for the MRI system; gradient coils configured toprovide gradient fields for the MRI system; and at least one RF coilconfigured to detect magnetic resonance (MR) signals; a controllerconfigured to: control the magnetics system to acquire MR spatialfrequency data using a non-Cartesian sampling trajectory; and generatean MR image from the acquired MR spatial frequency data using a neuralnetwork model comprising one or more neural network blocks including afirst neural network block, wherein the first neural network block isconfigured to perform data consistency processing using a non-uniformFourier transformation.

The foregoing is a non-limiting summary of the invention, which isdefined by the attached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects and embodiments of the disclosed technology will bedescribed with reference to the following figures. It should beappreciated that the figures are not necessarily drawn to scale.

FIG. 1A illustrates the architecture of an example neural network modelfor generating a magnetic resonance (MR) image from input MR spatialfrequency data, in accordance with some embodiments of the technologydescribed herein.

FIG. 1B illustrates the architecture of another example neural networkmodel for generating an MR image from input MR spatial frequency data,in accordance with some embodiments of the technology described herein.

FIG. 1C illustrates the architecture of yet another example neuralnetwork model for generating an MR image from input MR spatial frequencydata, in accordance with some embodiments of the technology describedherein.

FIG. 2A is a flowchart of an illustrative process 200 for generating anMR image from input MR spatial frequency data using a neural networkmodel, in accordance with some embodiments of the technology describedherein.

FIG. 2B is a flowchart of an illustrative process for processing MRspatial frequency data in the spatial frequency domain, which may bepart of the illustrative process 200, to obtain output spatial frequencydata, in accordance with some embodiments of the technology describedherein.

FIG. 2C is a flowchart of an illustrative process for processing spatialfrequency domain data, which may be part of the illustrative process200, to generate an MR image, in accordance with some embodiments of thetechnology described herein.

FIG. 2D is a flowchart of another illustrative process for processingimage domain data, which may be part of the illustrative process 200, togenerate an MR image, in accordance with some embodiments of thetechnology described herein.

FIG. 3 illustrates the performance of the techniques described hereinfor generating an MR image from input MR spatial frequency data using aneural network model having a locally-connected layer for operating ondata in the spatial frequency domain, in accordance with someembodiments of the technology described herein.

FIG. 4 illustrates the performance of the techniques described hereinfor generating an MR image from input MR spatial frequency data usingdifferent embodiments of the neural network model described herein.

FIG. 5A illustrates the architecture of another example neural networkmodel for generating a magnetic resonance (MR) image from input MRspatial frequency data, in accordance with some embodiments of thetechnology described herein.

FIG. 5B illustrates the architecture of another example neural networkmodel for generating a magnetic resonance (MR) image from input MRspatial frequency data, in accordance with some embodiments of thetechnology described herein.

FIG. 5C illustrates the architecture of another example neural networkmodel for generating a magnetic resonance (MR) image from input MRspatial frequency data, in accordance with some embodiments of thetechnology described herein.

FIGS. 6A-6C illustrate the distribution of weights of a fully-connectednetwork layer in a neural network sub-model configured to processspatial frequency domain data, in accordance with some embodiments ofthe technology described herein.

FIG. 7 illustrates results of generating MR images, from under-sampledspatial frequency domain data sampled using a non-Cartesian samplingtrajectory, using the techniques described herein and a zero-paddedinverse Fourier transform, in accordance with some embodiments of thetechnology described herein.

FIG. 8 illustrates aspects of training a neural network model forgenerating MR images from under-sampled spatial frequency domain data,in accordance with some embodiments of the technology described herein.

FIG. 9A illustrates aspects of generating synthetic complex-valuedimages for training a neural network model for generating MR images fromunder-sampled spatial frequency domain data, in accordance with someembodiments of the technology described herein.

FIG. 9B illustrates a loss function, having spatial frequency and imagedomain components, which may be used for training a neural network modelfor generating MR images from under-sampled spatial frequency domaindata, in accordance with some embodiments of the technology describedherein.

FIGS. 10A-10H illustrate reconstructed MR images using a zero-paddedinverse discrete Fourier transform (DFT) and using neural networkmodels, trained with and without transfer learning, in accordance withsome embodiments of the technology described herein.

FIG. 11 illustrates performance of some of the neural network models forgenerating MR images from under-sampled spatial frequency domain data,in accordance with some embodiments of the technology described herein.

FIG. 12 further illustrates performance of some of the neural networkmodels for generating MR images from under-sampled spatial frequencydomain data, in accordance with some embodiments of the technologydescribed herein.

FIG. 13A is a diagram of an illustrative architecture of an exampleneural network model for generating MR images from input MR spatialfrequency data, in accordance with some embodiments of the technologydescribed herein.

FIG. 13B is a diagram of one type of architecture of a block of theneural network model of FIG. 13A, in accordance with some embodiments ofthe technology described herein.

FIG. 13C is a diagram of an illustrative architecture of a dataconsistency block, which may be part of the block shown in FIG. 13B, inaccordance with some embodiments of the technology described herein.

FIG. 13D is a diagram of an illustrative architecture of a convolutionalneural network block, which may be part of the block shown in FIG. 13B,in accordance with some embodiments of the technology described herein.

FIG. 13E is a diagram of another type of architecture of a block of theneural network model of FIG. 13A, in accordance with some embodiments ofthe technology described herein.

FIG. 14 is a flowchart of an illustrative process 1400 for using aneural network model to generate an MR image from input MR spatialfrequency data obtained using non-Cartesian sampling, in accordance withsome embodiments of the technology described herein.

FIG. 15A illustrates T1-weighted MR images reconstructed by usingconventional neural network models and neural network models, inaccordance with some embodiments of the technology described herein.

FIG. 15B illustrates T2-weighted MR images reconstructed by usingconventional neural network models and neural network models, inaccordance with some embodiments of the technology described herein.

FIG. 15C illustrates reconstructed MR images at different stages ofprocessing by neural network models, in accordance with some embodimentsof the technology described herein.

FIG. 16 is a schematic illustration of a low-field MRI system, inaccordance with some embodiments of the technology described herein.

FIGS. 17A and 17B illustrate bi-planar permanent magnet configurationsfor a B₀ magnet, in accordance with some embodiments of the technologydescribed herein.

FIGS. 18A and 18B illustrate views of a portable MRI system, inaccordance with some embodiments of the technology described herein.

FIG. 18C illustrates a portable MRI system performing a scan of thehead, in accordance with some embodiments of the technology describedherein.

FIG. 18D illustrates a portable MRI system performing a scan of theknee, in accordance with some embodiments of the technology describedherein.

FIG. 19 is a diagram of an illustrative computer system on whichembodiments described herein may be implemented.

DETAILED DESCRIPTION

Conventional magnetic resonance imaging techniques require atime-consuming MRI scan for a patient in a tight chamber in order toobtain high-resolution cross-sectional images of the patient's anatomy.Long scan duration limits the number of patients that can be scannedwith MR scanners, causes patient discomfort, and increases the cost ofscanning. The inventors have developed techniques for generatingmedically-relevant, clinically-accepted MRI images from shorter-durationMRI scans, thereby improving conventional MRI technology.

The duration of an MRI scan is proportional to the number of data pointsacquired in the spatial frequency domain (sometimes termed “k-space”).Accordingly, one way of reducing the duration of the scan is to acquirefewer data points. For example, fewer samples may be acquired in thefrequency encoding direction, the phase encoding direction, or both thefrequency and phase encoding directions. However, when fewer data pointsare obtained than what is required by the spatial Nyquist criteria (thisis often termed “under-sampling” k-space), the MR image generated fromthe collected data points by an inverse Fourier transform containsartifacts due to aliasing. As a result, although scanning time isreduced by under-sampling in the spatial frequency domain, the resultingMRI images have poor quality and may be unusable, as the introducedartifacts may severely degrade image quality, fidelity, andinterpretability.

Conventional techniques for reconstructing MR images from under-sampledk-space data also suffer from drawbacks. For example, compressed sensingtechniques have been applied to the problem of generating an MR imagefrom under-sampled spatial frequency data by using a randomized k-spaceunder-sampling trajectory that creates incoherent aliasing, which inturn is eliminated using an iterative image reconstruction process.However, the iterative reconstruction techniques require a large amountof computational resources, do not work well without extensive empiricalparameter tuning, and often result in a lower-resolution MR image withlost details.

Deep learning techniques have also been used for reconstructing MRimages from under-sampled k-space data. The neural network parametersunderlying such techniques may be estimated using fully-sampled data(data collected by sampling spatial frequency space so that the Nyquistcriterion is not violated) and, although training such models may betime-consuming, the trained models may be applied in real-time duringacquisition because the neural network-based approach to imagereconstruction is significantly more computationally efficient than theiterative reconstruction techniques utilized in the compressive sensingcontext.

The inventors have recognized that conventional deep learning MR imagereconstruction techniques may be improved upon. For example,conventional deep learning MR image reconstruction techniques operateeither purely in the image domain or in the spatial frequency domainand, as such, fail to take into account correlation structure both inthe spatial frequency domain and in the image domain. As anotherexample, none of the conventional deep learning MR image reconstructiontechniques (nor the compressed sensing techniques described above) workwith non-Cartesian (e.g., radial, spiral, rosette, variable density,Lissajou, etc.) sampling trajectories, which are commonly used toaccelerate MRI acquisition and are also robust to motion by the subject.By contrast, the inventors have developed novel deep learning techniquesfor generating high-quality MR images from under-sampled spatialfrequency data that: (1) operate both in the spatial frequency domainand in the image domain; and (2) enable reconstruction of MR images fromnon-Cartesian sampling trajectories. As described herein, the deeplearning techniques developed by the inventors improve upon conventionalMR image reconstruction techniques (including both compressed sensingand deep learning techniques) and improve MR scanning technology byreducing the duration of scans while generating high quality MR images.

Some embodiments described herein address all of the above-describedissues that the inventors have recognized with conventional techniquesfor generating MR images from under-sampled spatial frequency domaindata. However, not every embodiment described below addresses every oneof these issues, and some embodiments may not address any of them. Assuch, it should be appreciated that embodiments of the technologyprovided herein are not limited to addressing all or any of theabove-described issues of conventional techniques for generating MRimages from under-sampled spatial frequency domain data.

Accordingly, some embodiments provide for a method of generating an MRimage from under-sampled spatial frequency domain data, the methodcomprising generating a magnetic resonance (MR) image from input MRspatial frequency data using a neural network model that comprises: (1)a first neural network sub-model configured to process spatial frequencydomain data; and (2) a second neural network sub-model configured toprocess image domain data. In this way, the techniques described hereinoperate both in the spatial-frequency and image domains.

In some embodiments, the first neural network sub-model is applied priorto the second neural network sub-model. In this way, a neural network isapplied to spatial-frequency domain data, prior to transforming thespatial-frequency domain data to the image domain, to take advantage ofthe correlation structure in the spatial frequency domain data.Accordingly, in some embodiments, generating the MR image may include:(1) processing the input MR spatial frequency data using the firstneural network sub-model to obtain output MR spatial frequency data; (2)transforming the output MR spatial frequency data to the image domain toobtain input image-domain data; and (3) processing the inputimage-domain data using the second neural network sub-model to obtainthe MR image.

In some embodiments, the first neural network sub-model may include oneor more convolutional layers. In some embodiments, one or more (e.g.,all) of the convolutional layers may have a stride greater than one,which may provide for down-sampling of the spatial-frequency data. Insome embodiments, the first neural network sub-model may include one ormore transposed convolutional layers, which may provide for up-samplingof the spatial frequency data. Additionally or alternatively, the firstneural network sub-model may include at least one locally-connectedlayer, at least one data consistency layer, and/or at least onecomplex-conjugate symmetry layer. In some embodiments, thelocally-connected layer may include a respective set of parameter valuesfor each data point in the MR spatial frequency data.

In some embodiments, the first neural network sub-model includes atleast one convolutional layer, a locally-connected layer, and at leastone transposed convolutional layer, and processing the input MR spatialfrequency data using the first neural network sub-model may include: (1)applying the at least one convolutional layer to the input MR spatialfrequency data; (2) applying the locally-connected layer to dataobtained using output of the at least one convolutional layer; and (3)applying the at least one transposed convolutional layer to dataobtained using output of the locally-connected layer. In suchembodiments, the first neural network sub-model may be thought of ashaving a “U” structure consisting of a down-sampling path (the left armof the “U”-implemented using a series of convolutional layers one ormore of which have a stride greater than one), a locally-connected layer(the bottom of the “U”), and an up-sampling path (the right arm of the“U”-implemented using a series of transposed convolutional layers).

In some embodiments, using a transposed convolutional layer (which issometimes termed a fractionally sliding convolutional layer or adeconvolutional layer) may lead to checkerboard artifacts in theupsampled output. To address this issue, in some embodiments, upsamplingmay be performed by a convolutional layer in which the kernel size isdivisible by the stride length, which may be thought of a “sub-pixel”convolutional layer. Alternatively, in other embodiments, upsampling toa higher resolution may be performed without relying purely on aconvolutional layer to do so. For example, the upsampling may beperformed by resizing the input image (e.g., using interpolation such asbilinear interpolation or nearest-neighbor interpolation) and followingthis operation by a convolutional layer. It should be appreciated thatsuch an approach may be used in any of the embodiments described hereininstead of and/or in conjunction with a transposed convolutional layer.

In some embodiments, the first neural network sub-model further takesinto account the complex-conjugate symmetry of the spatial frequencydata by including a complex-conjugate symmetry layer. In some suchembodiments, the complex-conjugate symmetry layer may be applied at theoutput of the transposed convolutional layers so that processing theinput MR spatial frequency data using the first neural network sub-modelincludes applying the complex-conjugate symmetry layer to data obtainedusing output of the at least one transposed convolutional layer.

In some embodiments, the first neural network sub-model further includesa data consistency layer to ensure that the application of first neuralnetwork sub-model to the spatial frequency data does not alter thevalues of the spatial frequency data obtained by the MR scanner. In thisway, the data consistency layer forces the first neural networksub-model to interpolate missing data from the under-sampled spatialfrequency data without perturbing the under-sampled spatial frequencydata itself. In some embodiments, the data consistency layer may beapplied to the output of the complex-conjugate symmetry layer.

In some embodiments, the first neural network sub-model includes aresidual connection. In some embodiments, the first neural networksub-model includes one or more non-linear activation layers. In someembodiments, the first neural network sub-model includes a rectifiedlinear unit activation layer. In some embodiments, the first neuralnetwork sub-model includes a leaky rectified linear unit activationlayer.

The inventors have also recognized that improved MR image reconstructionmay be achieved by generating MR images directly from spatial frequencydata samples, without gridding the spatial frequency data, as is oftendone in conventional MR image reconstruction techniques. In gridding,the obtained spatial frequency data points are mapped to atwo-dimensional (2D) Cartesian grid (e.g., the value at each grid pointis interpolated from data points within a threshold distance) and a 2Ddiscrete Fourier transform (DFT) is used to reconstruct the image fromthe grid values. However, such local interpolation introducesreconstruction errors.

The inventors have developed multiple deep-learning techniques forreconstructing MR images from data obtained using non-Cartesian samplingtrajectories. Some of the techniques involve using a non-uniform Fouriertransformation (e.g., a non-uniform fast Fourier transformation—NuFFT)at each of multiple blocks part of a neural network model in order topromote data consistency with the (ungridded) spatial frequency dataobtained by an MRI system. Such data consistency processing may beperformed in a number of different ways, though each may make use of thenon-uniform Fourier transformation (e.g., as represented by the forwardoperator A described herein), and the input MR spatial frequency data y.For example, in some embodiments, a non-uniform Fourier transformationmay be used in a neural network model block to transform image domaindata, which represents the MR reconstruction in the block, to spatialfrequency data so that the MR reconstruction in the block may becompared with the spatial frequency data obtained by the MRI system. Aneural network model implementing this approach may be termed thenon-uniform variational network (NVN) and is described herein includingwith reference to FIGS. 13A-13D.

As another example, in some embodiments, the non-uniform Fouriertransformation may be applied to the spatial frequency data, and theresult may be provided as input to each of one or more neural networkblocks of a neural network model for reconstructing MR images fromspatial frequency data. These innovations provide for a state-of-the artdeep learning technique for reconstructing MR images from spatialfrequency data obtained using a non-Cartesian sampling trajectory. Aneural network model implementing this approach may be termed thegeneralized non-uniform variational network (GNVN) and is describedherein including with reference to FIGS. 13A, 13D, and 13E.

Accordingly, some embodiments provide a method for generating a magneticresonance (MR) image from input MR spatial frequency data using a neuralnetwork model comprising one or more neural network blocks including afirst neural network block, wherein the first neural network block isconfigured to perform data consistency processing using a non-uniformFourier transformation (e.g., a non-uniform fast Fouriertransform—NuFFT) for transforming image domain data to spatial frequencydomain data. The MR spatial frequency data may have been obtained usinga non-Cartesian sampling trajectory, examples of which are providedherein. In some embodiments, the neural network model may includemultiple blocks each of which is configured to perform data consistencyprocessing using the non-uniform Fourier transformation.

In some embodiments, the method for generating the MR image from inputMR spatial frequency data includes: obtaining the input MR spatialfrequency data; generating an initial image from the input MR spatialfrequency data using the non-uniform Fourier transformation; andapplying the neural network model to the initial image at least in partby using the first neural network block to perform data consistencyprocessing using the non-uniform Fourier transformation.

In some embodiments, the data consistency processing may involveapplying a data consistency block to the data, which may apply anon-uniform Fourier transformation to the data to transform it from theimage domain to the spatial frequency domain where it may be comparedagainst the input MR spatial frequency data. In other embodiments, thedata consistency processing may involve applying an adjoint non-uniformFourier transformation to the input MR spatial frequency data andproviding the result as the input to each of one or more neural networkblocks (e.g., as input to each of one or more convolutional neuralnetwork blocks part of the overall neural network model).

In some embodiments, the first neural network block is configured toperform data consistency processing using the non-uniform Fouriertransformation at least in part by performing the non-uniform Fouriertransformation on data by applying a gridding interpolationtransformation, a fast Fourier transformation, and a de-apodizationtransformation to the data. In this way, the non-uniform Fouriertransformation A is represented as a composition of threetransformations—a gridding interpolation transformation G, a fastFourier transformation F_(s), and a de-apodization transformation D suchthat A=G F_(s) D, and applying A to the data may be performed byapplying the transformation D, F_(s), and G, to the data in that order(e.g., as shown in FIG. 13C). The gridding interpolation transformationmay be determined based on the non-Cartesian sampling trajectory used toobtain the initial MR input data. In some embodiments, applying thegridding interpolation transformation to the data may be performed usingsparse graphical processing unit (GPU) matrix multiplication. Examplerealizations of these constituent transformations are described herein.

In some embodiments, the neural network model to reconstruct MR imagesfrom spatial frequency data may include multiple neural network blockseach of which includes: (1) a data consistency block configured toperform the data consistency processing; and (2) a convolutional neuralnetwork block comprising one or more convolutional layers (e.g., havingone or more convolutional and/or transpose convolutional layers, havinga U-net structure, etc.). Such a neural network model may be termedherein as a non-uniform variational network (NVN).

In some embodiments, the data consistency block is configured to applythe non-uniform Fourier transformation to a first image, provided asinput to the data consistency block, to obtain first MR spatialfrequency data; and apply an adjoint non-uniform Fourier transformationto a difference between the first MR spatial frequency data and theinput MR spatial frequency data. In some embodiments, applying thenon-uniform Fourier transformation to the first image domain datacomprises: applying, to the first image domain data, a de-apodizationtransformation followed by a Fourier transformation, and followed by agridding interpolation transformation.

In some embodiments, applying the first neural network block to imagedomain data, the applying comprising: applying the data consistencyblock to image domain data to obtain first output; applying theplurality of convolutional layers to the image domain data to obtainsecond output; and determining a linear combination of the first andsecond output.

In some embodiments, the neural network model to reconstruct MR imagesfrom spatial frequency data may include multiple neural network blockseach of which includes a plurality of convolutional layers configured toreceive as input: (1) image domain data (e.g., representing the networkscurrent reconstruction of the MR data); and (2) output obtained byapplying an adjoint non-uniform Fourier transformation to the input MRspatial frequency data. Such a neural network model may be termed hereinas a non-uniform variational network (GNVN). In some embodiments, theplurality of convolutional layers is further configured to receive asinput: output obtained by applying the non-uniform Fouriertransformation and the adjoint non-uniform Fourier transformation to theimage domain data.

Another approach developed by the inventors for reconstructing an MRimage from input MR spatial frequency data, but without the use ofgridding, is to use at least one fully connected layer in the spatialfrequency domain. Accordingly, in some embodiments, the first neuralnetwork sub-model may include at least one fully connected layer that isto be applied directly to the spatial frequency data points obtained bythe scanner. The data points are not mapped to a grid (through griddingand/or any other type of local interpolation) prior to the applicationof the at least one fully connected layer. In some embodiments, the datapoints may be irregularly spaced prior to application of the at leastone fully connected layer.

In some of the embodiments in which the first neural network sub-modelincludes a fully-connected layer, the fully connected layer is appliedto the real part of the spatial frequency domain data, and the samefully-connected layer is applied to the imaginary part of the spatialfrequency domain data. In other words, the data is channelized and thesame fully connected layer is applied to both the real and imaginarydata channels.

Alternatively, in some of the embodiments in which the first neuralnetwork sub-model includes a fully connected layer, the first neuralnetwork sub-model includes a first fully-connected layer for applying tothe real part of the spatial frequency domain data and a secondfully-connected layer for applying to the imaginary part of the spatialfrequency domain data. In some embodiments, the first and secondfully-connected layers share at least some parameter values (e.g.,weights). In some embodiments, the output of the first and secondfully-connected layers is transformed using a Fourier transformation(e.g., a two-dimensional inverse discrete Fourier transformation) toobtain image-domain data. In turn, the image-domain data may be providedas input to the second neural network sub-model.

The mention of a 2D Fourier transformation in the preceding paragraphshould not be taken to imply that the techniques described herein arelimited to operating on two-dimensional data (e.g., on spatial frequencydomain and/or image domain data corresponding to a 2D MR image of abrain “slice”). In some embodiments, the techniques described herein maybe applied to 3D data (e.g., spatial frequency domain and/or imagedomain data corresponding to a stack of 2D MR images of differentrespective brain slices).

In some embodiments, batch normalization may be applied to the output offully-connected layer(s) prior to using the Fourier transformation toobtain image-domain data.

In some embodiments, the second neural network sub-model comprises atleast one convolutional layer and at least one transposed convolutionallayer. In some embodiments, the second neural network sub-modelcomprises a series of blocks comprising respective sets of neuralnetwork layers, each of the plurality of blocks comprising at least oneconvolutional layer and at least one transposed convolutional layer. Insome embodiments, each of the plurality of blocks further comprises: aFourier transformation layer, a data consistency layer, and an inverseFourier transformation layer.

In some embodiments, the neural network model used for generating MRimages from under-sampled spatial frequency data may be trained using aloss function comprising a spatial frequency domain loss function and animage domain loss function. In some embodiments, the loss function is aweighted sum of the spatial frequency domain loss function and the imagedomain loss function. In some embodiments, the spatial frequency domainloss function includes mean-squared error.

In some embodiments, the techniques described herein may be used forgenerating MR images from under-sampled spatial frequency data may beadapted for application to spatial frequency data collected using alow-field MRI system, including, by way of example and not limitation,any of the low-field MR systems described herein and in U.S. PatentApplication Publication No. “2018/0164390”, titled “ELECTROMAGNETICSHIELDING FOR MAGNETIC RESONANCE IMAGING METHODS AND APPARATUS,” whichis incorporated by reference herein in its entirety.

As used herein, “high-field” refers generally to MRI systems presentlyin use in a clinical setting and, more particularly, to MRI systemsoperating with a main magnetic field (i.e., a B₀ field) at or above 1.5T, though clinical systems operating between 0.5 T and 1.5 T are oftenalso characterized as “high-field.” Field strengths betweenapproximately 0.2 T and 0.5 T have been characterized as “mid-field”and, as field strengths in the high-field regime have continued toincrease, field strengths in the range between 0.5 T and 1 T have alsobeen characterized as mid-field. By contrast, “low-field” refersgenerally to MRI systems operating with a B₀ field of less than or equalto approximately 0.2 T, though systems having a B₀ field of between 0.2T and approximately 0.3 T have sometimes been characterized as low-fieldas a consequence of increased field strengths at the high end of thehigh-field regime. Within the low-field regime, low-field MRI systemsoperating with a B₀ field of less than 0.1 T are referred to herein as“very low-field” and low-field MRI systems operating with a B₀ field ofless than 10 mT are referred to herein as “ultra-low field.”

In order to train the neural network models described herein to generateMR images from (e.g., under-sampled) spatial frequency data obtained bya low-field MRI system, training data obtained using the low-field MRIsystem is needed. However, there are few low-field MRI scanners on themarket and little low-field MRI data available for training such neuralnetwork models. To address this limitation, the inventors have developeda novel two-stage training technique for training a neural network modelfor generating MR images from spatial frequency data obtained by alow-field MRI system. In the first stage, the neural network model(e.g., any of the neural network models described herein having a firstand a second neural network sub-model) is trained using a set of imagesobtained using a “high-field” or a “mid-field” MR system and,subsequently, be adapted by using a set of images obtained using alow-field MRI system.

Following below are more detailed descriptions of various conceptsrelated to, and embodiments of, methods and apparatus for generating MRimages from spatial frequency domain data. It should be appreciated thatvarious aspects described herein may be implemented in any of numerousways. Examples of specific implementations are provided herein forillustrative purposes only. In addition, the various aspects describedin the embodiments below may be used alone or in any combination, andare not limited to the combinations explicitly described herein.

FIG. 1A illustrates the architecture of an example neural network modelfor generating a magnetic resonance (MR) image from input MR spatialfrequency data, in accordance with some embodiments of the technologydescribed herein. As shown in FIG. 1A, the neural network model 100comprises first neural network sub-model 102 configured to processspatial frequency domain data, inverse fast Fourier transform (IFFT)layer 112 configured to transform spatial frequency domain data to imagedomain data, and second neural network sub-model 120 configured toprocess image domain data. After initial spatial frequency MR data isobtained using an MR scanner (e.g., using any of the low-field MRscanners described herein or any other suitable type of MR scanner), theinitial spatial frequency MR data may be processed using the firstneural network sub-model 102 to obtain output MR spatial frequency data111. The MR spatial frequency data 111 is then transformed by IFFT layer112 to obtain input image-domain data 113, which is processed by secondneural network sub-model 120 to obtain an MR image 127.

As shown in FIG. 1A, the first neural network sub-model 102 includes oneor more convolutional layers 104, a locally-connected layer 106, one ormore transposed convolutional layers 108, a residual connection 109,complex-conjugate symmetry layer 105 and a data consistency layer 110.

When the first neural network sub-model 102 is applied to initial MRspatial frequency data, the initial MR spatial frequency data is firstprocessed by one or more convolutional layers 104, then bylocally-connected layer 106, then by transposed convolutional layers108. In some embodiments the convolutional layer(s) 104 may be used todownsample the data and the transposed convolutional layers may be usedto upsample the data. In such embodiments, these three processing stepsmay be considered as providing a “U” shaped neural network architecture,with the convolutional layer(s) 104 providing a down-sampling path (leftarm of the “U”), the locally-connected layer 106 being at the bottom ofthe “U”, and the transposed convolutional layers 108 providing anup-sampling path (right arm of the “U”).

In the illustrated embodiment of FIG. 1A, the convolutional layer(s) 104include m₀ convolutional layers. In some embodiments, m0 may be 1, 2, 3,4, 5, or any number of layers between 1 and 20 layers. In someembodiments, one or more of the m0 convolutional layers may have astride greater than or equal to one. In some embodiments, one or more ofthe m0 convolutional layers has a stride greater than one, whichprovides for down-sampling or pooling the data through processing bysuch layers.

In the illustrated embodiment of FIG. 1A, the transposed convolutionallayer(s) 108 include m₀ transposed convolutional layers. In theillustrated embodiment of FIG. 1A, the number of convolutional layer(s)104 and the number of transposed convolutional layer(s) 108 is the same,but the number of convolutional and transposed convolutional layers maybe different in other embodiments.

In some embodiments, the locally-connected layer 106 is provided toexploit local correlation with K-space. In some embodiments, thelocally-connected layer 106 is not a convolutional layer (where the sameset of weights is applied across different portions of the data), butinstead has a respective set of weights for each data point in thespatial frequency domain data. In the illustrated embodiment of FIG. 1A,the locally-connected layer is placed between the down-sampling andup-samplings paths at the bottom of the “U” structure so that it wouldhave fewer parameters (since the resolution of the data is the lowest atthis point), which reduces the number of parameters that have to belearned during training.

In some embodiments, the locally-connected layer may account for energydensity variations in the spatial frequency domain (e.g., the centerregion in the spatial frequency domain has a higher energy density thanthe peripheral region). In the illustrative embodiment of FIG. 1A, thelocally-connected layer 106 operates in the spatial frequency domain andworks to interpolate the missing data (due to under-sampling) directlyin the spatial frequency domain. In practice, the locally-connectedlayer, which has far fewer parameters than a fully-connected layer, butmore parameters than convolutional layer, provides a good balancebetween training time and capability to interpolate the missing datapoints using the local contextual correlation of the spatial frequencydomain data.

It should be appreciated that using a locally-connected layer to accountfor energy density variations in the spatial frequency domain is a novelapproach developed by the inventors. Previous approaches split thespatial-frequency domain into three square regions, and the data in eachof the three regions was input into a separate model consisting of astack of convolutional layers (so three separate models for threedifferent square regions). By contrast, using a locally-connected layerdoes not involve partitioning k space into three square regions, andinstead involves assigning independent weights for each sign pixel,which accounts for the various energy density in a more general andflexible manner than previous approaches, resulting in a performanceimprovement.

FIG. 3 illustrates the performance improvement obtained by generating anMR image from input MR spatial frequency data using a neural networkmodel having a locally-connected layer. As can be seen in middle columnof FIG. 3, the MR image generated from a convolutional layer modelwithout a locally-connected layer generates artifacts (artificialstreaks) that deteriorate the image quality. By contrast, as shown inthe right column of FIG. 3, using a neural network model having asub-model with a locally-connected layer (e.g., locally connected layer106) eliminates such artifacts and produces an image closer to theoriginal image (left column of FIG. 3) in terms of mean-squared error.

Returning back to FIG. 1A, after data is processed by the layers 104,106, and 108, the data is provided to a complex-conjugate symmetry layer105, also termed the k-space symmetry layer, whose output is provided asinput to data consistency layer 110. The output of the data consistencylayer 110, which is also the output of the first neural networksub-model, is then provided as input to IFFT layer 112.

In some embodiments, the complex-conjugate symmetry layer 105 performsinterpolation based on the complex-conjugate symmetry in the spatialfrequency domain (whereby S(x, y)=S′(−x, −y) with (x,y) beingcoordinates of a data point and S′ representing the complex conjugationof S). In some embodiments, applying the complex-conjugate symmetrylayer 105 to spatial frequency domain data involves symmetricallymapping any missing points from existing samples. For example, if avalue were obtained for point (x,y), but no corresponding value wereobtained for point (−x,−y), the complex-conjugate symmetry layer may beused to provide the value for point (−x,−y) as the complex-conjugate ofthe obtained value for the point (x,y). Using the complex-conjugatesymmetry layer 105 accelerates the convergence of training the neuralnetwork model and improves the quality of images produces by the neuralnetwork model, as illustrated in the right panel of FIG. 4. Indeed, asshown in the right panel of FIG. 4, using the complex-conjugate symmetrylayer allows fewer training epochs to be used when training the neuralnetwork model while obtaining improved model performance, which ismeasured in this illustrative example by relative pixel intensityvariation in the center region of the images between the modelreconstructed image and the fully-sampled image.

In some embodiments, the data consistency layer 110 may be used toensure that the application of first neural network sub-model to thespatial frequency data does not alter the values of the spatialfrequency data obtained by the MR scanner. To the extent any such valuewas modified by other layers in the first neural network sub-model(e.g., by convolutional layer(s) 104, locally connected layer 106, andtransposed convolutional layer(s) 108), the modified values are replacedby the original values. In this way, the data consistency layer forcesthe first neural network sub-model to interpolate missing data from theunder-sampled spatial frequency data without perturbing theunder-sampled spatial frequency data itself.

In some embodiments, any of the neural network layers may include anactivation function, which may be non-linear. In some embodiments, theactivation function may be a rectified linear unit (ReLU) activationfunction, a leaky ReLU activation function, a hyperbolic tangent, asigmoid, or any other suitable activation function, as aspects of thetechnology described herein are not limited in this respect. Forexample, one or more of the convolutional layer(s) 104 may include anactivation function.

After the spatial frequency data is processed by the data consistencylayer 110, the data is provided as input to the IFFT layer 112, whichtransforms the spatial frequency data to the image domain—the output isinitial image domain data 113. The transformation may be performed usinga discrete Fourier transform, which may be implemented using a fastFourier transformation, in some embodiments. The initial image domaindata 113, output by the IFFT layer 112, is provided as input to thesecond neural sub-model 120.

As shown in FIG. 1A the second neural network sub-model 120 includesmultiple convolutional blocks 122, 124, and 126. Convolutional block 122may include one or more convolutional layers 128, an FFT layer 130, acomplex-conjugate symmetry layer 105, a data consistency layer, an IFFTlayer 134 and a residual connection. Each of the blocks 122, 124, and126 may have the same neural network architecture (e.g., these blocksmay have the same types of layers arranged in the same sequence), thoughthe various parameter values for the layers may vary (e.g., the weightsof the convolutional layers in block 122 may be different from that ofblock 124). Although in the illustrative embodiment of FIG. 1A, thesecond neural network sub-model 120 includes three convolutional blocks,this is by way of example, as in other embodiments the second neuralnetwork sub-model 120 may include any suitable number of convolutionalblocks (e.g., 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, or 15), asaspects of the technology described herein are not limited in thisrespect.

When the second neural network sub-model 120 is applied to initial imagedomain data 113 obtained at the output of the IFFT block 112, theconvolutional blocks 122, 124, and 126 are applied to initial imagedomain data 113 in that order. The application of convolutional block122 is described next, and it should be appreciated that theconvolutional blocks 124 and 126 may be applied in a similar way to theimage domain data provided as input to them.

As shown in FIG. 1A, convolutional block 122 includes at least oneconvolutional layer 128, followed by an FFT layer 130, acomplex-conjugate symmetry layer 105, data consistency layer 132, andIFFT layer 134.

In some embodiments, convolutional block 128 includes one or moreconvolutional layers with stride greater than 1 (e.g., 2 or greater) todownsample the image, followed by one or more transposed convolutionallayers with stride greater than 1 (e.g., 2 or greater), which upsamplethe image to its original size. This structure of down-sampling followedby up-sampling allows operations to be performed at differentresolutions, which helps the neural network model to capture both localand global features. In turn, this helps to eliminate image artifactsthat may result from under-sampling in the spatial frequency domain. Inthis illustrative embodiment, the convolutional layers do not includeskip connections, which may consume a substantial amount of memory. Forexample, in some embodiments, convolutional block 128 has five layerswith the number of filters being 16, 32, 64, 32, and 2, respectively. Insome embodiments, each of the filters may be a 3×3 filter with a LeakyReLU activation, though in other embodiments different size filtersand/or different activation functions may be used.

The impact of variable resolution layers is shown in FIG. 4, left panel.Indeed, as shown in the left panel of FIG. 4, using the variableresolution layers allows fewer training epochs to be used when trainingthe neural network model while obtaining improved model performance,which is measured in this illustrative example by relative pixelintensity variation in the center region of the images between the modelreconstructed image and the fully-sampled image.

As shown in the illustrative embodiment of FIG. 1A, after theconvolutional layers of convolutional block 122 are applied, the datamay be transformed into the spatial frequency domain so that thecomplex-conjugate symmetry and the data consistency blocks may beapplied, after which the data is transformed back into the image domain,and one or more other convolutional blocks may be applied.

In the embodiment illustrated in FIG. 1A, each of the convolutionalblocks 122, 124, and 126 includes complex-conjugate symmetry and dataconsistency blocks. However, in other embodiments, one or more (or all)of the convolutional blocks part of second neural network sub-model 120may not have either one or both of these blocks, as aspects of thetechnology described herein are not limited in this respect.

FIG. 1B illustrates the architecture of another example neural networkmodel 140 for generating MR images from input MR spatial frequency data,in accordance with some embodiments of the technology described herein.Neural network model 140 has a first neural network sub-model 142 with aconvolutional layer 146 instead of a locally-connected layer (e.g., incontrast with first neural network sub-model 102 of model 100 that has alocally connected layer 106). Such an embodiment may be advantageous asthe convolutional layer 142 has fewer parameters to learn duringtraining than the locally-connected layer 106. In other respects, neuralnetwork models 140 and 100 are the same.

FIG. 1C illustrates the architecture of yet another example neuralnetwork model 150 for generating MR images from input MR spatialfrequency data, in accordance with some embodiments of the technologydescribed herein. Neural network model 150 has a first neural networksub-model 152, with convolutional block 154 and transposed convolutionalblock 158. However, unlike corresponding convolutional block 104 andtransposed convolutional block 108 of neural network model 100, theconvolutional blocks 154 and 158 contain convolutional (and transposedconvolutional) layers using a stride of 1. As a result, the first neuralnetwork sub-model 152 does not perform up-sampling or down-sampling.Such an architecture may be advantageous when there is a large volume oftraining data available.

FIG. 2A is a flowchart of an illustrative process 200 for generating anMR image from input MR spatial frequency data using a neural networkmodel, in accordance with some embodiments of the technology describedherein. Process 200 may be implemented using any suitable neural networkarchitecture described herein including any of the neural networkarchitectures described with reference to FIGS. 1A-1C and 5A-5C. Process200 may be executed using any suitable computing device(s), as aspectsof the technology described herein are not limited in this respect. Forexample, in some embodiments, process 200 may be executed by a computingdevice communicatively coupled to or part of an MR imaging system.

Process 200 begins at act 202, where spatial frequency domain data isobtained. In some embodiments, the spatial frequency domain data may beobtained by using an MR scanner including any of the MR scannersdescribed herein. In other embodiments, the spatial frequency domaindata may have been obtained by an MR scanner prior to the execution ofprocess 200, stored, and the stored data may be accessed during act 202.

In some embodiments, the spatial frequency domain data may beunder-sampled relative to the Nyquist sampling criterion. For example,in some embodiments, the spatial frequency domain data may include lessthan 90% (or less than 80%, or less than 75%, or less than 70%, or lessthan 65%, or less than 60%, or less than 55%, or less than 50%, or lessthan 40%, or less than 35%, or any percentage between 25 and 100) of thenumber of data samples required by the Nyquist criterion.

The spatial frequency domain data obtained at act 202 may be (or mayhave been) obtained by an MR scanner using any suitable pulse sequenceand sampling scheme. For example, in some embodiments, the spatialfrequency domain data may be gathered using a Cartesian sampling scheme.In other embodiments, the spatial frequency domain data may be gatheredusing a non-Cartesian sampling scheme (e.g., radial, spiral, rosette,Lissajou, etc.).

Next, process 200 proceeds to act 204, where the MR spatial frequencydata obtained at act 202 is processed using a first neural networksub-model (e.g., sub-model 102 described with reference to FIG. 1A,sub-model 142 described with reference to FIG. 1B, sub-model 152described with reference to FIG. 1C, sub-model 502 described withreference to FIG. 5A, sub-model 522 described with reference to FIG. 5B,and sub-model 532 described with reference to FIG. 5C). Illustrativeexamples of how act 204 may be implemented are described with referenceto FIGS. 2B and 2C.

Next, process 200 proceeds to act 206, where the spatial frequencydomain data obtained at the completion of act 204 is transformed toobtain initial image domain data (e.g., using a Fourier transformation).

Next, process 200 proceeds to act 208, where initial the image domaindata obtained at the completion of act 206 is processed a second neuralnetwork sub-model (e.g., sub-model 120 described with reference to FIG.1A, sub-model 510 described with reference to FIG. 5A) to generate an MRimage. An illustrative example of how act 208 may be implemented isdescribed with reference to FIG. 2D.

FIG. 2B is a flowchart of an illustrative process for processing MRspatial frequency data in the spatial frequency domain, which may bepart of the illustrative process 200, to obtain output spatial frequencydata, in accordance with some embodiments of the technology describedherein. In particular, FIG. 2B shows an illustrative embodiment forimplementing act 204 of process 200.

As shown in FIG. 2B, act 204 may be implemented using acts 212-218. Atact 212, one or more convolutional layers may be applied to the spatialfrequency domain data obtained at act 202. In some embodiments, theconvolutional layer(s) applied at act 212 may be part of block 104described with reference to FIG. 1A or block 154 described withreference to FIG. 1C. In some embodiments, the convolutional layer(s)may include any suitable number of layers including any number of layersin the range of 1-20 layers. In some embodiments, the convolutionallayer(s) may be implemented using a stride greater than one (e.g., 2) todownsample the data. In other embodiments, the convolutional layer(s)may be implemented using a stride of 1.

Next, at act 214, a locally connected layer is applied to spatialfrequency domain data obtained at the completion of act 212. In someembodiments, the local convolutional layer may be the localconvolutional layer 106 described with reference to FIG. 1A. In someembodiments, the locally-connected layer has a respective set of weightsfor each data point in the spatial frequency domain data.

Next, at act 216, one or more transposed convolutional layers areapplied to spatial frequency domain data obtained at the completion ofact 214. In some embodiments, the transposed convolutional layer(s) maybe the transposed convolutional layer(s) part of block 108 describedwith reference to FIG. 1A or block 158 described with reference to FIG.1C. In some embodiments, the transposed convolutional layer(s) mayupsample the data.

Next, at act 218, a complex conjugate symmetry layer is applied to thespatial frequency domain data output at the completion of act 216. Insome embodiments, the complex conjugate symmetry layer may be thecomplex conjugate symmetry layer 105 described with reference to FIG.1A. As described herein, applying the complex-conjugate symmetry layer105 to spatial frequency domain data may involve symmetrically mappingany missing points from existing samples. For example, if a value wereobtained for point (x,y), but no corresponding value were obtained forpoint (−x,−y), the complex-conjugate symmetry layer may be used toprovide the value for point (−x,−y) as the complex-conjugate of theobtained value for the point (x,y).

Next, at act 220, a data consistency layer is applied to the spatialfrequency domain data output at the completion of act 218. In someembodiments, the data consistency layer may be the data consistencylayer 110 described with reference to FIG. 1A. As described herein, thedata consistency layer may be used to ensure that the application offirst neural network sub-model to the spatial frequency data does notalter the values of the spatial frequency data obtained by the MRscanner.

FIG. 2C is a flowchart of an illustrative process for processing spatialfrequency data, which may be part of the illustrative process 200, togenerate an MR image, in accordance with some embodiments of thetechnology described herein. In particular, FIG. 2C shows anotherillustrative embodiment for implementing act 204 of process 200.

As shown in FIG. 2C, act 204 may be implemented using acts 222 and 224.At act 222, one or more fully connected layers are applied to thespatial frequency data obtained at act 202. In some embodiments, thefully connected layer(s) applied at act 222 may be fully connected layer502 described with reference to FIG. 5A. As described herein, the fullyconnected layer represents a learned mapping from non-Cartesian toCartesian coordinates from data, which allows MR images to bereconstructed from non-Cartesian samples without relying on conventionalgridding or other interpolation schemes, which are not data dependent.

In some embodiments, at act 222, the spatial frequency data obtained atact 202 is split into real and imaginary portions and the same fullyconnected layer is applied to each of the two portions. Equivalently,one may consider these data as being provided to a fully connected layerwith shared weights for the real and imaginary channels. Such a weightsharing scheme ensures that the same interpolation operation is appliedto both the real and imaginary channels, which preserves the underlyingspatial frequency domain symmetry throughout the process. In addition,sharing the weights between the real and imaginary portions reduces thenumber of trainable parameters in the model by a factor of two. However,in other embodiments, the spatial frequency data may be fed to a fullyconnected layer with partial or no weight sharing between the real andimaginary channels.

Next, at act 224, batch normalization is applied so that the subsequentlayer receives input having a substantially 0 mean and a substantiallyunit (or any other suitable constant) variance.

It should be appreciated that the process of FIG. 2C is illustrative andthat there are variations. For example, in some embodiments, the batchnormalization may be omitted.

FIG. 2D is a flowchart of another illustrative process for processingimage-domain data, which may be part of the illustrative process 200, togenerate an MR image, in accordance with some embodiments of thetechnology described herein. In particular, FIG. 2D shows anillustrative embodiment for implementing act 208 of process 200.

As shown in FIG. 2D, act 208 may be implemented using acts 230-236 anddecision block 238. In particular, at act 230, one or more convolutionallayers are applied to image domain data obtained at act 206 bytransforming spatial frequency domain data to the image domain. In someembodiments, the convolutional layer(s) applied at act 230 may be partof block 128 shown in FIG. 1A or block 512 shown in FIG. 5A. In someembodiments, the convolutional layer(s) may include any suitable numberof layers including any number of layers in the range of 1-20 layers. Insome embodiments, the convolutional layer(s) may be implemented using astride greater than one (e.g., 2) to downsample the data. In otherembodiments, the convolutional layer(s) may be implemented using astride of 1.

Next, at act 232, one or more transposed convolutional layers may beapplied to the image-domain data output at the completion of act 230. Insome embodiments, the transposed convolutional layer(s) applied at act232 may be part of transpose block 128 shown in FIG. 1A or block 512shown in FIG. 5A. In some embodiments, the convolutional layer(s) mayinclude any suitable number of layers including any number of layers inthe range of 1-20 layers. In some embodiments, the transposedconvolutional layer(s) may be implemented to upsample the data (e.g.,using a fractional stride).

Next, at act 234, a complex-conjugate symmetry layer may be applied tothe data. As the complex-conjugate symmetry layer is applied in thespatial frequency domain, the image domain data output at the completionof act 232 is transformed to the spatial frequency domain prior to theapplication of the complex-conjugate symmetry layer. In someembodiments, the complex conjugate symmetry layer may be thecomplex-conjugate symmetry layer 105 described with reference to FIG.1A.

Next, at act 236, a data consistency layer may be applied to the data.In some embodiments, the data consistency layer may be applied tospatial frequency domain data output at completion of act 234. In otherembodiments, if act 234 were omitted, the image domain data output atthe completion of act 232 may be transformed to the spatial frequencydomain and the data consistency layer may be applied thereto. In someembodiments, the data consistency layer may be the data consistencylayer 110 described with reference to FIG. 1A.

Next, at decision block 238, a determination is made as to whether oneor more additional image-domain processing blocks are to be applied.When it is determined that no further blocks are to be applied, theprocess completes. Otherwise, the process returns to act 230, via the“YES” branch, and acts 230-236 and decision block 238 are repeated. Forexample, as shown in FIG. 1A, after block 122 is applied to the imagedomain data, it may be determined that block 124 is to be applied to thedata.

It should be appreciated that the process of FIG. 2D is illustrative andthat there are variations. For example, in some embodiments, theimage-domain data may be processed purely in the image domain withoutapplication of the complex-conjugate symmetry layer and the dataconsistency layer.

FIG. 5A illustrates the architecture of another example neural networkmodel 500 for generating a magnetic resonance (MR) image from input MRspatial frequency data, in accordance with some embodiments of thetechnology described herein.

As shown in FIG. 5A, the neural network model 500 comprises first neuralnetwork sub-model 502 configured to process spatial frequency domaindata, inverse fast Fourier transform (IFFT) layer 508 configured totransform spatial frequency domain data to image domain data, and secondneural network sub-model 510 configured to process image domain data.After initial spatial frequency MR data is obtained using an MR scanner(e.g., using any of the low-field MR scanners described herein or anyother suitable type of MR scanner), the initial spatial frequency MRdata may be processed using the first neural network sub-model 502 toobtain output MR spatial frequency data 511. The MR spatial frequencydata 511 is then transformed by IFFT layer 508 to obtain initialimage-domain data 513, which is processed by second neural networksub-model 510 to obtain an MR image 518.

As shown in FIG. 5A, the initial spatial frequency domain MR data issplit into a real portion 504 (e.g., magnitudes of the complex-valueddata) and imaginary portion 506 (e.g., phases of the complex-valueddata). The first neural network sub-model 502 includes a fully connectedlayer that operates on the real portion 504 and imaginary portion 506.In the embodiment shown in FIG. 5A, the fully connected layer sharesweights between the real and imaginary channels. As such, the fullyconnected layer applies the same operations to both the real andimaginary channels, which preserves the underlying spatial frequencydomain symmetry throughout the process. In addition, sharing the weightsbetween the real and imaginary portions reduces the number of trainableparameters in the model by a factor of two. However, in otherembodiments (e.g., the embodiment of FIG. 5C), the spatial frequencydata may be fed to a fully connected layer with partial or no weightsharing between the real and imaginary channels.

In some embodiments, when the neural network model including thefully-connected layer is trained using input MR images generated usingthe same sample trajectory, the fully-connected layer learns adata-dependent mapping from non-Cartesian to Cartesian coordinates,which can be used to perform a data-dependent gridding of non-Cartesianspatial-frequency data that may be generated by an MR scanner operatingin accordance with a non-Cartesian sequence. This is illustrated furtherin FIGS. 6A-6C.

FIG. 6A shows an illustrative embodiment in which each data point in thespatial frequency domain has a corresponding 128×128 weight matrixhaving a weight for each location in a 128×128 output k-space, creatinga non-local interpolation. The distribution of weights for three spatialfrequency domain data points (#300, #2800, and #5000) is shown in FIG.6B. The 2D distributions of these same three data points are shown inFIG. 6C, with zoomed-in views to show the details of the weightdistribution.

As shown in the 1D and 2D weight distributions of FIGS. 6B-6C, whenplotting a two-dimensional weight map of a particular spatial frequencydomain data point, it is predominantly the weights in a localneighborhood of the data point that have non-negligible values, withother weights having values close to zero. The weight distributionindicates that the mapping performed by the fully-connected layerperforms a local interpolation. It should be noted that the first neuralnetwork sub-model 502 does not include a data consistency layer, whichallows the first neural network sub-model 502 to process non-Cartesiansamples.

Returning to FIG. 5A, after the spatial frequency data is processed bythe first neural network model 502, the data is provided as input to theIFFT layer 508, which transforms the spatial frequency data to the imagedomain—the output is initial image domain data 513. The transformationmay be performed using a discrete Fourier transform, which may beimplemented using a fast Fourier transformation, in some embodiments.The initial image domain data 513, output by the IFFT layer 508, isprovided as input to the second neural sub-model 510.

As shown in FIG. 5A the second neural network sub-model 510 includesmultiple convolutional blocks 512, 514, and 516. Convolutional block 512may include one or more convolutional layers and a residual connection.Each of the convolutional blocks 512, 514, and 516 may have the sameneural network architecture (e.g., these blocks may have the same typesof layers arranged in the same sequence), though the various parametervalues for the layers may vary (e.g., the weights of the convolutionallayers in block 512 may be different from that of block 514). Althoughin the illustrative embodiment of FIG. 5A the second neural networksub-model 510 includes three convolutional blocks, this is by way ofexample, as in other embodiments the second neural network sub-model 510may include any suitable number of convolutional blocks (e.g., 1, 2, 4,5, 6, 7, 8, 9, 10, 11, 12, 13, 14, or 15), as aspects of the technologydescribed herein are not limited in this respect.

When the second neural network sub-model 510 is applied to initial imagedomain data 513 obtained at the output of the IFFT block 508, theconvolutional blocks 512, 514, and 516 are applied to initial imagedomain data 513 in that order. The application of convolutional block512 is described next, and it should be appreciated that theconvolutional blocks 514 and 516 may be applied in a similar way to theimage domain data provided as input to them (after being output from thepreceding block).

In some embodiments, convolutional block 512 includes one or moreconvolutional layers with stride greater than 1 (e.g., 2 or greater) todownsample the image, followed by one or more transposed convolutionallayers with stride greater than 1 (e.g., 2 or greater), which upsamplethe image to its original size. This structure of down-sampling followedby up-sampling allows operations to be performed at differentresolutions, which helps the neural network model to capture both localand global features. In turn, this helps to eliminate image artifactsthat may result from under-sampling in the spatial frequency domain.

For example, in some embodiments, convolutional block 512 may includetwo sequential convolutional layers (having 32 3×3 and 64 3×3 filters inthe two respective layers, with stride 2), followed by two transposedconvolutional layers (128 3×3 and 64 3×3 filters in the two respectivelayers, with stride 2), followed by a final convolutional layer (2 3×3filters with stride 1). A non-linear activation (e.g., a ReLU or a LeakyReLU activation) may be applied in each of the first four layers, exceptfor the final convolutional layer. Though, it should be appreciated thatin other embodiments, different size filters and/or different activationfunctions may be used, as aspects of the technology described herein arenot limited in this respect.

FIG. 5B illustrates the architecture of another example neural networkmodel 520 for generating a magnetic resonance (MR) image from input MRspatial frequency data, in accordance with some embodiments of thetechnology described herein. Neural network 520 has a first neuralnetwork sub-model 522 with a batch normalization layer 507 followingapplication of the fully connected layer and prior to the output of datafrom the first neural network sub-model 522 to the IFFT layer 508.Introducing a batch normalization layer at this juncture improves theperformance of the neural network and may reduce the time required fortraining. In other respects, neural network models 520 and 500 are thesame.

FIG. 5C illustrates the architecture of another example neural networkmodel 530 for generating a magnetic resonance (MR) image from input MRspatial frequency data, in accordance with some embodiments of thetechnology described herein. Neural network 530 has a first neuralnetwork sub-model 532 which includes a fully connected layer that doesnot use weight sharing between the real and imaginary portions of theobtained MR data. In other respects, neural network models 530 and 500are the same.

The inventors have developed a novel non-Cartesian sampling trajectoryto accelerate acquisition of spatial domain data, while retaining asmuch information as possible. The sampling trajectory consists ofunstructured triangular and tetrahedral meshes to evenly under-samplethe entire spatial frequency domain, and a fully sampling grid in thek-space center generated by a Gaussian kernel, as full coverage of thek-space center is important for reconstructions of images with lowsignal-to-noise ratio (SNR). This sampling trajectory samples 33% of thespatial frequency domain samples need to satisfy the Nyquist criterion(though as described above a sampling trajectory may be used with anyother percentage described herein, including for example any percentagein the range of 25-100, such as 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%,75%, 80%, etc.). K-space. FIG. 7 illustrates the novel non-Cartesiansampling trajectory (panel 702), the image reconstructed from samplesobtained using the trajectory of panel 702 and a zero-padded inversefast Fourier transform (panel 704), the image reconstructed from samplesobtained using the trajectory of panel 702 and the neural network modeldescribed with reference to FIG. 5B (panel 706), and the original MRimage. As can be seen from panels 704 and 706, the MR image obtainedusing a zero-padded IFFT is blurred and has artifacts, while the MRimage obtained using the neural network model of FIG. 5B does not sufferfrom these drawbacks.

The inventors have developed specialized techniques for training theneural network models described herein. The training procedure involvesgenerating complex image data, under-sampling the complex image data,and using pairs of under-sampled and fully sampled complex image data totrain the neural network model using any suitable training techniques(e.g., stochastic gradient descent and back-propagation). In order togenerate complex image data, magnitude images were used to synthesizethe phase information, as described below.

FIG. 8 illustrates aspects of training a neural network model forgenerating MR images from under-sampled spatial frequency domain data,in accordance with some embodiments of the technology described herein.As shown in FIG. 8, the training process involves using input magnitudeimages to synthesize the phase information. The magnitude and phaseinformation that constitute the complex image data which can beretrospectively under-sampled in the spatial frequency domain usingCartesian or a non-Cartesian (e.g., radial. etc.) sampling trajectory.The under-sampled data will be used as the input to the neural networkmodel being trained, while the full-sampled image will be the output ofthe model.

Although there are many publicly available MR image datasets available,they typically only include magnitude images. To simulate complex dataas acquired by an MR scanner, the inventors have developed a techniquefor generating phase information to append to the magnitude images.Accordingly, in some embodiments, phase information is generated using aweighted sum of spherical harmonic basis functions. The combination ofthese functions can characterize magnetic field variation derived frominhomogeneity of the B₀, magnetic field drifting with temperature,gradient eddy currents, spatially-varying RF coil sensitivity fields,inaccuracies in gradient fields in sequences and/or other effects thatmay contribute to phase variation. The process of generating phaseinformation using spherical harmonics is illustrated in FIG. 9A.

In some embodiments, to simulate non-Cartesian under-sampling, anon-uniform FFT (NuFFT) was used to transform MR images to thespatial-frequency domain where a non-Cartesian under-sampling mask wasapplied. In turn, the under-sampled spatial frequency data can beconverted to the image domain using an inverse (also called backward)NuFFT, which can be provided as input to the image-domain sub-models. Inthis way, the use of NuFFT, enables performing non-uniform K-spacesampling, which highly resembles the non-Cartesian sampling in practice.

In some embodiments, the available training data was augmented byapplying affine transformations to individual slices to create imageswith different orientation and size, adding noise to create images withdifferent SNR, introducing motion artifacts, incorporating phase and/orsignal modulation for more complex sequences like echo trains, and/ormodeling the dephasing of the signal to adapt the model to a sequencelike diffusion weighted imaging.

As the neural network models described herein operate both in thespatial frequency domain and in the image domain, the inventors havedeveloped a new loss function to facilitate training such a mixed-domainneural network model. The new loss function accelerated the process oftraining the neural network models described herein (e.g., by reducingthe number of training epochs needed to achieve a given level ofperformance).

In some embodiments, the loss function includes a first loss function tocapture error in the spatial frequency domain and a second loss functionto capture error in the image domain. For example, as shown in FIG. 9B,the output of the first neural network sub-model (labeled as “Subnet 1k-Space”) may be compared to ground truth in the spatial frequencydomain to obtain a first measure of error (e.g., mean squared error,labeled “MSE Loss 1”) in the spatial frequency domain, and the output ofthe second neural network sub-model (labeled as “Subnet 2 Image domain”)may be compared to ground truth in the image domain to obtain a secondmeasure of error (e.g., mean squared error, labeled “MSE Loss 2”) in theimage domain. The first and second measures of error may be combined(e.g., via a weighted combination) to produce an overall measure oferror, which is to be minimized during the training process. Forexample, in the illustrative example of FIG. 9, the two loss functionswere combined using a weight of X<1 such that the overall loss functionwas given by Loss1+X*Loss2.

As described herein, in order to train the neural network modelsdeveloped by the inventors to generate MR images from under-sampledspatial frequency data obtained by a low-field MRI system, training dataobtained using the low-field MRI system is needed. However, there maynot be a sufficient volume of such data to learn all the parameters ofthe models described herein.

Accordingly, in some embodiments, a neural network model is firsttrained using images obtained using one or more “high-field” and/or a“mid-field” MR systems and then transfer learning is used to adapt thetrained neural network model to the “low-field” context by using one ormore MR images obtained using a low-field MRI system.

FIGS. 10A-10H illustrates MR images generated using a zero-paddedinverse DFT and using neural network models, trained with and withouttransfer learning, in accordance with some embodiments of the technologydescribed herein. The results show that using transfer learning (100epochs in this illustrative example) improves performance of the modelon low-field MR images. In particular, FIG. 10A-10D show reconstructedMR images obtained, respectively, using a zero-padded inverse FFT, theneural network model of FIG. 5B trained without transfer learning, theneural network of FIG. 5B trained with transfer learning, as well as thefully sampled data. The FIGS. 10E-10G show the absolute differencebetween the reconstructed MR images and the fully sampled MR images,while FIG. 10H shows the under-sampling mask.

FIG. 11 illustrates performance of some of the neural network models forgenerating MR images from under-sampled spatial frequency domain data,in accordance with some embodiments of the technology described herein.In particular, the second row of FIG. 11 shows the performance of theneural network model 100 described with reference to FIG. 1A, and thethird row of FIG. 11 shows the performance of the neural network model520 described with reference to FIG. 5B. For both models, FIG. 11 showsthe performance of the respective first and second sub-models(sub-models 102 and 120, and sub-models 522 and 510). The first row ofFIG. 11 shows the under-sampled and fully-sampled images (both magnitudeand phase). As may be seen from FIG. 11, the output of the firstsub-model of the neural network model 100 (first two columns in themiddle row) has improved quality with fewer artifacts, which is alsoindicated by the increased peak SNR (PSNR). The output of the secondsub-model of the neural network model 100 (last two columns in themiddle row) shows that the second sub-model further improves thereconstruction by increasing the contrast of the magnitude image andgenerating a smoother phase map, which is closer to that of the fullysampled image. For the neural network model 520, the second sub-modelcontributes less to the improvement as reflected by PSNR than the firstsub-model. The situation is reversed for the first neural networksub-model.

FIG. 12 further illustrates performance of some of the neural networkmodels for generating MR images from under-sampled spatial frequencydomain data, in accordance with some embodiments of the technologydescribed herein. In particular, FIG. 12 illustrates the performance ofsome of the neural networks developed herein relative to othertechniques on images under-sampled down to 33% of the number of samplesrequired by the Nyquist sampling rate. The performance of the neuralnetwork models 100 and 520 (shown in fourth and fifth columns of FIG.12) was compared to that of compressed sensing (implemented using theADMM regularizer, with regularization parameter=5e-3, and shown in thesecond column of FIG. 12) and neural network model 100 without the firstsub-model (shown in the third column of FIG. 12). Normalized meansquared error and peak-SNR were measured to quantify the difference ofoutput images. As shown in the FIG. 12, under-sampling introducesblurring and inhomogeneous artifacts. The compressed sensing approachremoves the artifacts, but over-smooths the image, and alters the phaseimage. The model 100 without its first sub-model failed to recover theimage. By contrast, the neural network models 100 and 520, output MRimages that are much closer in both magnitude and phase to the fullysampled image, as reflected by higher PSNR and lower normalized MSE thancompeting methods.

As discussed herein, the inventors have developed neural network modelsfor reconstructing MR images from spatial frequency data obtained usingnon-Cartesian sampling trajectories.

FIG. 13A is a diagram of an illustrative architecture of an exampleneural network model 1310 for generating MR images from input MR spatialfrequency data, in accordance with some embodiments of the technologydescribed herein. As shown in FIG. 13A, neural network model 1310reconstructs output MR image 1315 from input MR spatial frequency data1305 by processing the input MR spatial frequency data in stages. First,the input MR spatial frequency data 1305 is processed using initialprocessing block 1312 to produce an initial image 1314, and then theinitial image 1314 is processed by a series of neural network blocks1316-1, 1316-2, . . . , 1316-n.

In some embodiments, one or more of the blocks 1316-1, 1316-2, . . . ,1316-n may operator in the image domain. In some embodiments, one ormore of the blocks 1316-1, 1316-2, . . . , 1316-n may transform theinput data to a different domain, including but not limited to thespatial frequency domain, perform processing (e.g., reconstructionprocessing) in the different domain, and subsequently transform back tothe image domain.

In some embodiments, the initializer block transforms the input MRspatial frequency data to the image domain to generate an initial imagefor subsequent processing by the neural network model 1310. Theinitializer block may be implemented in any suitable way. For example,in some embodiments, the initializer block may apply the adjointnon-uniform Fourier transformation to the input MR spatial frequencydata to obtain the initial image. As another example, in someembodiments, the initializer block may apply the gridding reconstructionto the input MR spatial frequency data to obtain the initial image.

Illustrative architectures of neural network blocks 1316 are shown inFIG. 13B (corresponding to a non-uniform variational network) and FIG.13E (corresponding to a generalized non-uniform variational network).Accordingly, in some embodiments, at least one, at least some, or all ofthe blocks 1316-1, 1316-2, . . . , 1316-n may have an architecture asshown for illustrative block 1316-i in FIG. 13B. As shown in FIG. 13-B,neural network block 1316-i includes a data consistency block 1320, anda convolutional neural network block 1350, both of which are applied tothe input x_(i), labeled as 1321. The input x_(i) may represent the MRimage reconstruction generated by neural network 1310 at the completionof the (i−1)^(st) neural network block. In this example, the output 1335of the block 1316-i is obtained by applying the data consistency block1320 to the input x_(i), to obtain a first result, applying theconvolutional neural network block 1350 to x_(i), to obtain a secondresult, and subtracting from x_(i) a linear combination of the firstresult and the second result, where the linear combination is calculatedusing the block-specific weight λ_(i).

The data consistency block 1320 may be implemented in any of numerousways. In some embodiments, the data consistency block 1320 may performdata consistency processing by transforming the input image representedby x_(i) to the spatial frequency domain using a non-uniform Fouriertransformation, comparing the result with the initial MR spatialfrequency data 1305, and transforming the difference between the twoback to the image domain using an adjoint of the non-uniform Fouriertransformation.

An illustrative implementation of data consistency block 1320 is shownin FIG. 13C. In the illustrative implementation of FIG. 13C, the imagedomain input 1322 (which may be the intermediate reconstruction x_(i)1321), is transformed to the spatial frequency domain through a seriesof three transformations 1324, 1326, and 1328, whose composition is usedto implement a non-uniform fast Fourier transformation from the imagedomain to the spatial frequency domain. In particular, thetransformation 1324 is a de-apodization and zero-padding transformationD, the transformation 1326 is an oversampled FFT transformation F_(s),and the transformation 1328 is the gridding interpolation transformationG. As described herein, the non-uniform fast Fourier transformation A isrepresented by the composition of these transformations according to:A=D F_(s) G. Example realizations of these constituent transformationsare described herein.

After the image domain input 1322 is transformed to the spatialfrequency domain, it is compared with the initial MR spatial frequencydata 1305, and the difference between the two is transformed back to theimage domain using the transformations 1330, 1332, and 1334, in thatorder. The transformation 1330 is the adjoint of the griddinginterpolation transformation 1328. The transformation 1332 is theadjoint of the oversampled FFT transformation 1326. The transformation1334 is the adjoint of the deapodization transformation 1324. In thisway, the composition of the transformations 1330, 1332, 1334, which maybe written as G^(H)F^(H) _(s) D^(H)=A^(H), represents the adjoint A^(H)of the non-uniform Fourier transformation A.

The convolutional neural network block 1350 may be implemented in any ofnumerous ways. In some embodiments, the block 1350 may have multipleconvolutional layers, including one or more convolutional layers and oneor more transpose convolutional layers. In some embodiments, the block1350 may have a U-net structure, whereby multiple convolutional layersdownsample the data and subsequent transpose convolutional layersupsample the data, for example, as shown in the illustrative U-netarchitecture of FIG. 13D for the block 1350.

As shown in FIG. 13D, input to the convolutional network block 1350 isprocessing by a downsampling path followed an upsampling path. In thedownsampling path, the input is processed by repeated application of twoconvolutions with 3×3 kernels, each followed by application of anon-linearity (e.g., a rectified linear unit or ReLU), an average 2×2pooling operation with stride 2 for downsampling. At each downsamplingstep the number of feature channels is doubled from 64 to 128 to 256. Inthe upsampling path, the data is processed be repeated upsampling of thefeature map using an average unpooling step that halves the number offeature channels, a concatenation with the corresponding feature mapfrom the downsampling path, and two 3×3 convolutions, each followed byapplication of a non-linearity (e.g., a ReLU).

FIG. 13E is a diagram of another type of architecture of a block of theneural network model of FIG. 13A, in accordance with some embodiments ofthe technology described herein. A neural network model with blockshaving the architecture like the one shown in FIG. 13E may be termed a“generalized non-uniform variational network” or “GNVN”. It is“generalized” in the sense that, while data consistency blocks are notused directly, feature similar to the image features generated by suchblocks may be useful to incorporate into a neural network model.

As shown in FIG. 13E, the i^(th) GNVN block 1360-i takes as input: (1)the image domain data x_(i), labeled as 1362; and (2) the initial MRspatial frequency data 1364. The input x_(i) may represent the MR imagereconstruction generated by neural network 1310 at the completion of the(i−1)^(st) GNVN block (1360-(i−1)). These inputs to the block 1360-i arethen used to generate inputs to the convolutional neural network block1372 part of block 1360-i. In turn, from these inputs, the CNN block1372 generates the next MR image reconstruction denoted by x_(i+1).

In the embodiment of FIG. 13E, the inputs 1362 and 1364 are used togenerate three inputs to the CNN block 1372: (1) the reconstructionx_(i) itself is provided as input to the CNN block; (2) the result ofapplying, to the reconstruction x_(i), the non-uniform Fouriertransformation 1366 followed by a spatial frequency domain convolutionalneural network 1368, followed by the adjoint non-uniform Fouriertransformation 1370; and (3) the result of applying, to the initial MRspatial frequency data 1364, the spatial frequency domain convolutionalneural network 1368 followed by an adjoint non-uniform Fourier transform1370.

In some embodiments, the non-uniform Fourier transformation 1366 may bethe transformation A expressed as a composition of threetransformations: the de-apodization transformation D, an oversampledFourier transformation F_(s), and a local gridding interpolationtransformation G such that A=D F_(s) G. Example realizations of theseconstituent transformations are described herein.

The spatial frequency domain CNN 1368 may be any suitable type ofconvolutional neural network. For example, the CNN 1368 may be a fivelayer convolutional neural network with residual connection. However, inother embodiments, the spatial frequency domain network 1368 may be anyother type of neural network (e.g., a fully convolutional neuralnetwork, a recurrent neural network, and/or any other suitable type ofneural network), as aspects of the technology described herein are notlimited in this respect.

A discussion of further aspects and details of neural network models forMR image reconstruction from non-Cartesian data, such as the neuralnetwork models illustrated in FIGS. 13A-13E, follows next. First, somenotation is introduced. Let x ∈

^(N) denote a complex-valued MR image to be reconstructed, representedas a vector with N=N_(x)N_(y) where N_(x) and N_(y) are width and heightof the image. Let y∈

^(M) (M<<N) represent the undersampled k-space measurements from whichthe complex-valued MR image x is to be reconstructed. Reconstruct x fromy may be formulated as an unconstrained optimization problem accordingto:

$\begin{matrix}{{{\underset{x}{argmin}\frac{\lambda}{2}{{{Ax} - y}}_{2}^{2}} + {{`}(x)}},} & \left( {{Eq}.\mspace{11mu} 1} \right)\end{matrix}$

where the operator A is a non-uniform Fourier sampling operator,

expresses regularisation terms on x, and λ is a hyper-parameterassociated to the noise level. In the case when the k-space measurementsy are obtained using a Cartesian sampling trajectory, the operator A mayexpressed according to: A=MF where M is a sampling mask, and F isdiscrete Fourier transform. In the case of a non-Cartesian samplingtrajectory, the measurements no longer fall on a uniform k-space gridand the sampling operator A is now given by a non-uniform discreteFourier transform of type I:

$\begin{matrix}{{y\left( \left( {k_{x},k_{y}} \right) \right)} = {\sum\limits_{l = 0}^{N_{x}}{\sum\limits_{m = 0}^{N_{y}}{x_{lm}{e^{2\pi \; i}\left( {{\frac{l}{N_{x}}k_{x}} + {\frac{m}{N_{y}}k_{y}}} \right)}}}}} & \left( {{Eq}.\mspace{11mu} 2} \right)\end{matrix}$

where (k_(x), k_(y))∈

² (rather than (k_(x), k_(y))∈

²). An efficient implementation of the above forward model may beimplemented using the so-called non-uniform Fast Fourier Transform(NUFFT). The idea is to approximate Eq. 2 by the followingdecomposition: A=GF_(s)D, where G is a gridding interpolation kernel,F_(s) is fast Fourier transform (FFT) with an oversampling factor of s,and D is a de-apodization weights. This decomposition is described infurther detail below.

In contrast, the inversion of A is considerably more involved. For the(approximately) fully-sampled case, one can consider direct inversion (

(N³)) or a more computationally efficient gridding reconstruction, whichhas the form X_(gridding)=A^(H) Wy, where W is a diagonal matrix usedfor the density compensation of non-uniformly spaced measurements. Forthe undersampled case, the inversion is ill-posed, and Eq. 1 should besolved by iterative algorithms.

The inventors have developed a new deep learning algorithm toapproximate the solution to the optimization problem of Eq. 1. Theapproach begins by considering a gradient descent algorithm, whichprovides a locally optimal solution to Eq. 1, specified by the followingequations for initialization and subsequent iterations:

x ₀ =f _(init)(A,y);  (Eq. 3)

x _(i+1) =x _(i)−α_(i)∇_(x) f(x)_(x=x) _(i) ,  (Eq. 4)

where f_(init) is an initializer, α is a step size and ∇f is thegradient of the objective functional, which is given by:

∇_(x) f(x)=λA ^(H)(Ax−y)+∇_(x)

R(x).  (Eq. 5)

In some embodiments, the initializer may be selected as the adjointf_(init) (A, y)=A^(H)y reconstruction or the gridding reconstructionf_(init) (A, y)=A^(H)Wy. The deep learning approach to solving Eq. 1involves unrolling the sequential updates of Eq. 4 into a feed-forwardmodel, and approximating the gradient term ∇

by a series of trainable convolutional (or other types of neuralnetwork) layers and non-linearities. This approach results in anend-to-end trainable network with N_(it) blocks given by:

$\begin{matrix}{x_{0} = {f_{{init}\text{-}{cnn}}\left( {A,\left. y \middle| \theta_{0} \right.} \right)}} & \left( {{Eq}.\mspace{11mu} 6} \right) \\{x_{i + 1} = {x_{i} - {\lambda_{i}\underset{\underset{D\; C\text{-}i}{}}{A^{H}\left( {{Ax}_{i} - y} \right)}} - \underset{\underset{{CNN}\text{-}i}{}}{f_{nn}\left( x_{i} \middle| \theta_{i} \right)}}} & \left( {{Eq}.\mspace{11mu} 7} \right)\end{matrix}$

where the learnable parameters are {θ₀, . . . , θ_(N) _(it) , λ₁, . . ., λ_(N) _(it) }. Note that the step size a₁ is absorbed in the learnableparameters. In this way, a general non-convex regularization functionalis used (e.g., instead of a Fields-of-Experts model), and regularizationfunctional can be approximated by convolutional neural networks. Indeed,the neural network model illustrated in FIGS. 13A-13D is implemented inaccordance with Equations 6 and 7. For example, an implementation of thedata consistency term DC-i is shown in FIG. 13C and an implementation ofthe CNN-i term is shown in FIG. 13D.

The inventors have recognized that the computational complexity of suchan approach is a function of how the forward operator A∈

^(M×N) is implemented because A is large complex-valued matrix that canoccupy a lot of memory to store. As described herein, in contrast to theCartesian case, A is expressed as GF_(s)D. For 2D cases, this can be alarge matrix, which consumes a large portion of GPU memory (e.g., forN=192² and M=10,000 (i.e., ≈3×acceleration), storing the complex-valuedmatrix alone already takes 3 GB of memory). To overcome this challenge,the inventors have implemented the gridding interpolation transformationG i as a sparse GPU matrix multiplication. F_(s) is an FFT, where anefficient GPU implementation is available. Finally, D is a diagonalmatrix, which can be implemented as a Hadamard product of matrices. Theadjoint can similarly be implemented as A^(H)=D^(H) F_(s) ^(H) G^(H),where ⋅^(H) is a complex-conjugate transpose.

Further details of the decomposition of the forward operator A=GF_(s)Dare described next. First, some preliminaries. The spatial frequencydomain (sometimes referred to as k-space) may be indexed usingtwo-dimensional or three-dimensional coordinates (e.g. (k_(x), k_(y)) or(k_(x),k_(y),k_(z))). In this way, each entry of the vector yrepresenting input MR spatial frequency data represents a valueassociated to a specific coordinate in k-space. A regular grid ink-space refers to a regularly-spaced grid of points k-space such thatthere is a fixed distance Δ between each k-space coordinate that can beindexed. Generally, the input MR spatial frequency data y may includek-space samples spaced on a regular-grid or irregularly spaced.Regularly spaced points are sometimes termed Cartesian data points.Irregularly spaced points are sometimes termed non-Cartesian (data)points.

The interpolation transformation G operates to interpolate non-Cartesiansensor data y onto a regular k-space grid. When the transformation isrepresented as a matrix G, each row in the matrix corresponds to aspecific regular grid point in k-space, and the entry j in the row i(i.e., the entry G_(ij)) expresses how much weight is associated betweenith regular grid and jth k-space sample.

In some embodiments, the interpolation matrix entries may be computedany one of the following four functions:

${{Two}\mspace{14mu} {term}\mspace{14mu} {cosine}\mspace{14mu} \alpha} + {\left( {1 - \alpha} \right){\cos \left( {\frac{2\pi}{W}u} \right)}}$${{Three}\text{-}{term}\mspace{14mu} {cosine}\text{:}\mspace{14mu} \alpha} + {\beta \; {\cos \left( {\frac{2\pi}{W}u} \right)}} + {\left( {1 - \alpha - \beta} \right){\cos \left( {\frac{4\pi}{W}u} \right)}}$${Gaussian}\text{:}\mspace{14mu} {\exp \left\lbrack {{- \frac{1}{2}}\left( \frac{u}{\sigma} \right)^{2}} \right\rbrack}$${Kaiser}\text{-}{Bessel}\text{:}\mspace{14mu} \frac{1}{W}{I_{0}\left\lbrack {\beta \sqrt{1 - \left( {2{u/W}} \right)^{2}}} \right\rbrack}$

where u is a distance between ith regular grid point and jthnon-Cartesian data coordinate. The parameters α, β, W, σ are free designparameters to be specified by user, and I₀ is the zeroth-order modifiedBessel function of the first kind. However, it should be appreciatedthan any other function may be used for computing the interpolationmatrix entries instead of or in addition to the example four functionslisted above.

In some embodiments, the entries of the interpolation weight matrix maybe computing using an optimization approach. For example, the entriesmay be computed by solving a min-max optimization problem, as shown inEquations 16 and 21 of Fessler, J. A., Sutton B. P.: Non-uniform fastFourier transforms using min-max interpolation. IEEE Transactions ofSignal Processing 51(2), 560-574 (2003), which is incorporated byreference herein in its entirety. In some embodiments, the Fouriertransformation F may be represented by an oversampled Fourier matrixF_(s), which is a dense matrix in which each entry is a complexexponential of the form e^(iγ) for γ which depends on the index. Therole of this matrix is to perform Fourier transform. In someembodiments, F_(s) may be implemented using the fast Fourier transformwith oversampling factor s. For example, if the image to bereconstructed x is N×N pixels, then oversampling FFT is performed forimage size sN×sN.

In some embodiments, the de-apodization transformation may berepresented by a matrix D that will weigh each pixel in the image by acorresponding weight to reduce the interpolation error of approximatingA with the given decomposition. In some embodiments, this may beimplemented via a pixel-wise weighting of the intermediatereconstruction in the image domain. For example, the pixel-wiseweighting may be implemented using a spatially-varying low-order smoothpolynomial. In some embodiments, the matrix D may be set as discussed inSection IV-C of Fessler, J. A., Sutton B. P.: Non-uniform fast Fouriertransforms using min-max interpolation. IEEE Transactions of SignalProcessing 51(2), 560-574 (2003).

The inventors have also appreciated that the network of FIGS. 13A-13Dforces a bottleneck at the end of each iteration. However, analternative view is that the network simply benefits from the imagefeatures given by data consistency (DC-i) blocks. This observationmotivates a generalized approach where, instead of using a dataconsistency block, each CNN-i block in the model of FIGS. 13A-13D isprovided a concatenation of the following inputs: the intermediatereconstruction x_(i), the self-adjoint A^(H)Ax_(i), and the adjoint ofthe input A^(H) y. Furthermore, one can also consider applying1D-convolution in raw sensory domain using f_(sensor-cnn)(Ψ|ϕ) toexploit the information along the sampling trajectory and removeunnecessary information (e.g. isolatable artifacts or noise). Theresulting network, shown in FIGS. 13A, 13D, and 13E, is given by:

x ₀ =f _(init-cnn)(A,f _(sensor-cnn)(y|ϕ ₀)|θ₀)x _(i+1)

f _(cnn)(x _(i) ,A ^(H) f _(sensor-cnn)(Ax _(i)|ϕ_(i)),x ₀|θ_(i),),

where the learnable parameters are {ϕ₀, . . . , ϕ_(N) _(it) , θ₀, . . ., θ_(N) _(it) }. As described herein, this type of neural network modelis termed Generalized Non-uniform Variational Network (GNVN).

The inventors have recognized that some embodiments of neural networkarchitectures described herein may be considered as embodiments of aneural network model that may be expressed according to the following:

x _(rec) =f _(rec)(A,y|θ)  (Eq. 8),

This general type of neural network model may accepts as input any inputthat is a combination of the forward operator A and raw spatialfrequency domain data y, as well as additional learnable parameters θ,which can be an arbitrary dimension. The parameters θ may be adjustedduring training process.

The input to the neural network of Eq. 8 may be data obtained by one ormultiple RF coils of an MRI system, as aspects of the technologydescribed herein are not limited to reconstructing images from datacollected by a single RF coil. In addition, the input data y may havebeen obtained using multiple contrasts and/or different sets ofacquisition parameters (e.g., by varying repetition time (TR), echo time(TE), flip angle θ, etc.). In some embodiments, input into the networkmay be, but is not limited to, the raw data y. Additionally oralternatively, the input to the network may be the adjointreconstruction A^(H) y where (⋅)^(H) is the conjugate transpose of thematrix.

In some embodiments, where the data y includes data collected bymultiple RF coils, these data y may be split into N_(coil) separate datasets, denoted y^((i)) for i=1, . . . , N_(coil). N_(coil) can be anynumber (e.g., any number in the range of 2-20 such, for example, 8 or 9or 10). In some such embodiments, the neural network input may be theadjoint reconstruction of each coil images x₀ ^((i))=A^(H) y^((i)), andx₀ ^((i)) for i=1, . . . , N_(coil) can be stacked together and form theinput to the network (e.g., to the convolutional layers part of thenetwork).

In some embodiments, the raw data y may include multiple measurementsobtained by each of one or more RF coils. For example, if the data ismeasured multiple times, say N_(avg) times, then these data, or theadjoint reconstruction of these data, or any other function of thesedata measurements and the forward operator A, may form an input to theneural network. For example, multiple measurements may be obtained forsignal averaging and/or as part of acquiring images with differentcontrast.

In some embodiments, as described above, the input to the neural networkof Eq. 8 may be also be any function based on A and/or y. For example,in some embodiments, the gridding reconstruction may be an input to thenetwork. Gridding reconstruction may have the form of x₀=A^(H)Wy, whereW is called sample density compensation weights, which is a matrix thatscales each element in the vector y.

Any of numerous techniques may be used to compute the sample densitycompensation weights W. For example, in some embodiments, the weights Wmay be computed according to: W=A^(H)A1, where 1 is a vector of ones. Asanother example, the weights W may be any suitable user-definedfunction. As yet another example, the weights W may be learned andadjusted during neural network training, in which case the weights maybe referred to as learned sample density compensation weights. In someembodiments, the input to the network may be a combination of y and theweights W, whether learned or fixed learnable, without the use of theforward operator A.

It should also be appreciated that the neural network need not operateon the raw data y, and in some embodiments these data may bepre-processed. For example, in some embodiments these data may bepre-processed to perform operations such as interference removal,denoising, filtering, smoothing, image prewhitening, etc. Moregenerally, the network has the form f (y, A, θ).

With regard to the neural network weights θ, these weights may beinitialized in any suitable way as part of the training procedure. Forexample, the weights may be initialized randomly (e.g., using Heinitialization following Equation 12 in He, K., et al.: Deep residuallearning for image recognition. Proceedings of the IEEE conference oncomputer vision and pattern recognition (CVPR). pp. 1026-1034 (2015)).As another example, the network weights may be initialized according toa setting provided by a user. As another example, the network weightsmay include the learned sampling density weights (e.g., the learnedsampling density weights may be a subset of the network weights, thenetwork weights may be initialized using the learned sampling densityweights, and all the weights may subsequently be adjusted duringtraining).

With regard to the output x_(rec) of the neural network in Eq. 8, theoutput may include one or more images per respective RF coil. Forexample, if the input data contains data from each of N_(coil) RF coils,the output may include one MR image for each such RF coil or multiple MRimages for each such coil (e.g., when each coil performs multipleacquisitions, for example, using different contrasts).

In some embodiments, multiple neural networks of the type specified inEq. 8 may be employed and the output of these networks may be combinedsuch that the multiple neural networks are utilized as an ensemble. Thecombination may be performed using any suitable type of aggregation ruleincluding, but not limited to, average, weighted averaging, averagingwith outlier rejection, and selection of the “best” reconstructionaccording to a user-defined criterion (e.g., manual inspection,automated selection based on a quantitative metric such as the signal tonoise ratio, a perceptual metric, and/or any other suitable metric).Alternatively, in some embodiments, multiple instances of x_(rec) fromindividual neural networks may be stacked together, and be considered asthe output of the network.

As described above, there are numerous possible embodiments of theneural network formulation of Eq. 8 including, but not limited to, theembodiments described herein such as: (1) the non-uniform variationalnetwork (NVN) as described herein including with reference to FIGS.13A-D; (2) the generalized non-uniform variational network (GNVN) asdescribed herein with reference to FIGS. 13A, 13D, and 13E; (3) the DeepK-space Interpolation Reconstruction (DKIR) network as described hereinincluding with reference to FIGS. 1A-C; and (4) the Deep non-localreconstruction (DNR) network as described herein including withreference to FIGS. 5A-5C.

It should be noted that while some of the above described networksarchitectures include convolutional neural network blocks, other typesof blocks may be used in addition to or instead of the convolutionalneural network blocks including, for example, residual network, denselyconnected networks, or squeeze and excitation networks.

In some embodiments, any one of the networks described above may betrained using mean-squared error. For example, in some embodiments, eachof the reconstruction blocks in the NVN (e.g., blocks 1316-i) or GNVN(e.g., blocks 1360-i) architectures may be trained using the meansquared-error criterion according to:

${\mathcal{L}(\theta)} = {\sum\limits_{{({y,x})}^{\in }}{{x - x_{rec}}}_{2}}$

In some embodiments, a reconstruction block can reconstruct eachcoil-weighted images x_(c) separately or jointly. It can also attempt toreconstruct each signal n_(avg)=1, . . . , N_(avg) jointly orseparately.

FIG. 14 is a flowchart of an illustrative process 1400 for using aneural network model to generate an MR image from input MR spatialfrequency data obtained using non-Cartesian sampling, in accordance withsome embodiments of the technology described herein. In someembodiments, process 1400 may be performed using a non-uniformvariational network (e.g., the neural network described with referenceto FIGS. 13A-D), a generalized non-uniform variation network (e.g., theneural network described with reference to FIGS. 13A, 13D, and 13E), orany other suitable type of neural network model.

In some embodiments, the illustrative process 1400 may be performedusing any suitable computing device. For example, in some embodiments,the process 1400 may be performed by a computing device co-located(e.g., in the same room as) with an MRI system that obtained the inputMR spatial frequency data by imaging a subject. As another example, insome embodiments, the process 1400 may be performed by one or moreprocessors located remotely from the MRI system (e.g., as part of acloud computing environment) that obtained the input spatial frequencydata by imaging a subject.

Process 1400 begins at act 1402, where input MR spatial frequency datais obtained. In some embodiments, the input MR spatial frequency datahad been previously obtained by an MRI system and stored for subsequentanalysis, so that it is accessed at act 1402. In other embodiments, theinput MR spatial frequency data may be obtained by an MRI system(including any of the MRI systems described herein) as part of process1400. Regardless of when an MRI system performed the imaging to obtainthe input MR spatial frequency data, the data may have been obtainedusing a non-Cartesian sampling trajectory, examples of which areprovided herein.

Next, process 1400 proceeds to act 1404, where the input MR spatialfrequency data may be pre-processed to obtain an initial imagereconstruction. For example, in some embodiments, the input MR spatialfrequency data may be transformed to the image domain by using anon-uniform Fourier transformation. For example, the input MR spatialfrequency data y may be transformed to the image domain using theadjoint operator A^(H) described herein (e.g., by computing A^(H)y). Asanother example, the input MR spatial frequency data may be transformedto the image domain using a gridding reconstruction such as A^(H)Wy,where the matrix W is a sampling density compensation matrix that couldbe: the matrix A^(H)A1, where 1 is a vector of one's, a user-specifiedmatrix, a matrix learned during training, and/or any suitablecombination thereof. In the illustrative example of FIG. 13A, thepre-processing may be performed by the initial processing block 1312.

In some embodiments, the initializer block transforms the input MRspatial frequency data to the image domain to generate an initial imagefor subsequent processing by the neural network model 1310. Theinitializer block may be implemented in any suitable way. For example,in some embodiments, the initializer block may apply the adjointnon-uniform Fourier transformation to the input MR spatial frequencydata to obtain the initial image. As another example, in someembodiments, the initializer block may apply the gridding reconstructionto the input MR spatial frequency data to obtain the initial image.

Next, process 1400 proceeds to act 1406, where a block of a neuralnetwork model is applied to the initial image obtained at act 1404 (orto the current image data when act 1406 is being executed on a returnpath from decision block 1408 after one or more neural network blockshave already been applied to the initial image). In some embodiments,the block of the neural network model may be configured to perform dataconsistency processing by using a non-uniform Fourier transformation totake into account the initial MR spatial frequency data obtained at act1402. This may be done in any suitable way. For example, in someembodiments, the data consistency processing may be performed by a dataconsistency block such as block 1316-i described with reference to FIG.13B. In such a block, data consistency processing involves transformingintermediate reconstructions transformed to the spatial frequency domainusing a non-uniform Fourier transformation and comparing the result tothe input MR spatial frequency data. As another example, in someembodiments, the data consistency processing may be performed bytransforming the input MR spatial frequency data to the image domainusing the non-uniform Fourier transformation and providing the result asinput to one or more convolutional blocks as is done, for example, inneural network block 1360-i described with reference to FIG. 13E.

Next, process 1400 proceeds to decision block 1408 where it isdetermined whether another neural network block is to be applied. If itis determined that another block is to be applied, process 1400 returnsto act 1406, where another neural network block is applied to the imagedata generated at the completion of the last iteration of block 1406.Otherwise, this image data is output as the final reconstructed MR imageat act 1410.

The inventors have evaluated the performance of the neural networkarchitectures described herein including with reference to FIGS. 13A-Eand 14 on real-world MR images. The details of these experiments aredescribed next.

As part of the experiments, 640 randomly selected T1-weighted andT2-weighted brain images were obtained from Human Connectome Project(https:///www.humanconnectome.org/study/hcp-young-adult/document/1200-subjects-data-release).Six hundred of the images were used for training the neural network,while 40 of the images were used for evaluating the performance of thetrained neural network. To perform a realistic simulation, a number ofpre-processing steps were performed. First, since only magnitude imageswere provided from the Human Connectome Project, complex-valued imageswere created by adding phase information to the magnitude data usingtwo-dimensional Fourier bases with randomly sampled low ordercoefficients. Second, the images were multiplied by spatially localizedcomplex coil sensitivity profiles, which was derived from an analyticalmodel of an MRI RF coil. Finally, a realistic amount of noise observablefor parallel image acquisition was added to the images. For theexperiments, the images were resampled to a field of view (FOV) of180×180×180 mm³, with the isotrophic resolution of 3.4×3.4×3.4 mm³,1.7×1.7×1.7 mm³ and 1.15×1.15×1.15 mm³, resulting in the matrix sizes64³, 128³ and 192³, respectively.

In these experiments, single coil reconstruction is evaluated in orderto study the behavior of non-uniform MR data reconstruction. The MR datawas under-sampled using 2D non-uniform variable density, where thesampling density decays from the k-space center at quadratic speed. Foreach matrix size, the sampling trajectory with the target accelerationfactor R∈{2,4} was generated. For evaluation, we measured mean squarederror (MSE), structural similarity index measurement (SSIM), and peaksignal-to-noise ratio (PSNR).

The techniques developed herein were developed with a number ofconventional techniques that have been applied to non-uniform MR dataincluding: (1) AUTOMAP (Zhu B., et al.: Image reconstruction bydomain-transform manifold learning. Nature 555(7697), 487 (2018)); (2)image domain U-net (Han, Y., et al.: Deep learning with domainadaptation for acceleration projection-reconstruction MR. Magneticresonance in medicine 80(3), 118-1205 (2018)); and (3) k-space domainU-net. Id. All deep learning methods were trained using MSE. Due to itshigh GPU memory requirements, AUTOMAP was trained only for the matrixsize of 64×64. For the NVN approach having the architecture shown inFIGS. 13A-D, a U-net with 3 levels of downsampling (see e.g., FIG. 13D)for each convolutional sub-block. N_(it)=5 blocks was used for thenumber of blocks, and the adjoint A^(H)y was used for f_(init). For theGNVN approach, a 5-layer convolutional neural network was usedf_(sensor-cnn). Each network was trained for 8,000 epochs using Adamoptimizer with α=10⁻⁴, β=0.9, β₂=0.999. All methods were implemented inTensorFlow.

Results of the evaluations are summarized in Table 1 below. The NVN andGNVN approaches developed by the inventors consistently outperformed thebaseline approaches for both acceleration factors. AUTOMAP and k-spaceU-net both underperformed compared to other methods.

TABLE 1 Quantitative result for acceleration factor (R) 2 and 4. Foreach metric, mean and standard deviation is computed. For mean squarederror (MSE), the values are scaled by 10³. R = 2 R = 4 Methods MSE SSIMPSNR MSE SSIM PSNR 64 × 64 AUTOMAP 2.40 (42.14) 0.87 (0.14) 29.87 (3.73)2.59 (8.09) 0.84 (0.14) 28.36 (3.51) 64 × 64 U-net 1.53 (18.13) 0.92(0.11) 31.44 (3.86) 2.25 (21.87) 0.90 (0.10) 29.81 (3.74) 64 × 64 U-net(k) 1.91 (7.40) 0.86 (0.13) 30.07 (3.57) 2.51 (6.58) 0.81 (0.13) 28.48(3.34) 64 × 64 NVN 1.22 (12.51) 0.93 (0.11) 32.33 (3.92) 1.38 (4.04)0.92 (0.09) 30.95 (3.62) 64 × 64 GNVN 1.22 (16.88) 0.93 (0.09) 32.54(4.00) 1.37 (4.58) 0.92 (0.08) 31.08 (3.66) 128 × 128 U-net 0.75 (3.73)0.94 (0.09) 34.06 (3.68) 0.91 (4.10) 0.94 (0.07) 32.76 (3.50) 128 × 128U-net (k) 1.02 (1.26) 0.89 (0.10) 32.51 (3.58) 1.54 (13.77) 0.87 (0.11)31.32 (3.48) 128 × 128 NVN 0.57 (0.86) 0.95 (0.06) 34.68 (3.57) 0.82(1.07) 0.93 (0.07) 32.95 (3.54) 128 × 128 GNVN 0.58 (1.99) 0.95 (0.07)34.83 (3.64) 0.67 (0.79) 0.95 (0.03) 33.65 (3.47) 192 × 192 U-net 0.47(1.55) 0.96 (0.05) 35.68 (3.67) 0.67 (1.13) 0.94 (0.07) 33.71 (3.23) 192× 192 U-net (k) 0.77 (0.81) 0.89 (0.10) 33.83 (3.62) 1.31 (7.53) 0.87(0.11) 31.84 (3.35) 192 × 192 NVN 0.40 (0.60) 0.96 (0.06) 36.11 (3.60)0.66 (1.40) 0.91 (0.12) 34.01 (3.43) 192 × 192 GNVN 0.40 (0.77) 0.96(0.05) 36.15 (3.57) 0.52 (0.44) 0.96 (0.03) 34.36 (3.07)

As between the NVN and GNVN approaches, while the NVN approach showedhigher data fidelity (lower mean-squared error), the GNVN approachoffered better values for PSNR and SSIM. The sample reconstructions ofT1-weighted image for R=2 and T2-weighted image for R=4 is shown in FIG.15A and FIG. 15B respectively. While the overall differences betweenU-net, NVN and GNVN were small, the reconstructions from NVN and GNVNresulted in lower error, owing to the data consistency processing. GNVNresulted in the lowest overall errors and preserved more of the finedetails. Nevertheless, a certain level of blurriness can be observed inall images, due to the added noise. Again, U-net (k-space) for singlecoil resulted in a suboptimal reconstruction qualitatively. In FIG. 15C,we visualize the output of NVN and GNVN at each block. Interestingly,unlike compressed sensing methods, the intermediate image can divergefrom the final image. This is unsurprising as there was no constraint toenforce such behavior. For NVN, most output of each block seems closerto the ground truth, presumably because the output of the DC-i and CNN-iblocks are explicitly combined. In comparison, GNVN showed moreinteresting features for all the intermediate stages, mainlyhighlighting the high frequency information.

In these experiments, the number of parameters were 128.1M, 22.0M, 6.6Mand 7.3M for AUTOMAP (64×64), U-net, NVN and GNVN respectively. Thereconstruction speed were 5.928±0.020 ms, 19.145±0.072 ms, 19.459±0.077ms, 44.934±0.088 ms, and 65.520±0.100 ms for AUTOMAP (for the image size64³), U-net, U-net (k-space), NVN and GNVN respectively for the imagesize 192³.

FIG. 16 is a block diagram of exemplary components of a MRI system 1600.In the illustrative example of FIG. 16, MRI system 1600 comprisesworkstation 1604, controller 1606, pulse sequences store 1608, powermanagement system 1610, and magnetic components 1620. It should beappreciated that system 1600 is illustrative and that an MRI system mayhave one or more other components of any suitable type in addition to orinstead of the components illustrated in FIG. 16.

As illustrated in FIG. 16, magnetic components 1620 comprises B₀ magnet1622, shim coils 1624, RF transmit and receive coils 1626, and gradientcoils 1628. B₀ magnet 1622 may be used to generate, at least in part,the main magnetic field B₀. B₀ magnet 1622 may be any suitable type ofmagnet that can generate a main magnetic field (e.g., a low-fieldstrength of approximately 0.2 T or less), and may include one or more B₀coils, correction coils, etc. Shim coils 1624 may be used to contributemagnetic field(s) to improve the homogeneity of the B₀ field generatedby magnet 1622. Gradient coils 1628 may be arranged to provide gradientfields and, for example, may be arranged to generate gradients in themagnetic field in three substantially orthogonal directions (X, Y, Z) tolocalize where MR signals are induced.

RF transmit and receive coils 1626 may comprise one or more transmitcoils that may be used to generate RF pulses to induce a magnetic fieldB₁. The transmit/receive coil(s) may be configured to generate anysuitable type of RF pulses configured to excite an MR response in asubject and detect the resulting MR signals emitted. RF transmit andreceive coils 1626 may include one or multiple transmit coils and one ormultiple receive coils. The configuration of the transmit/receive coilsvaries with implementation and may include a single coil for bothtransmitting and receiving, separate coils for transmitting andreceiving, multiple coils for transmitting and/or receiving, or anycombination to achieve single channel or parallel MRI systems. Thus, thetransmit/receive magnetic component is often referred to as Tx/Rx orTx/Rx coils to generically refer to the various configurations for thetransmit and receive component of an MRI system.

Each of magnetics components 1620 may be of any suitable type and may beconstructed in any suitable way. For example, in some embodiments, theB₀ magnet 1622 may be an electromagnet or a permanent magnet (e.g., asdescribed below with reference to FIGS. 17A-B and 18A-B). As anotherexample, in some embodiments, one or more magnetics components 1620(e.g., shim coils 1624 and/or gradient coils 1628) may be fabricatedusing the laminate techniques.

Power management system 1610 includes electronics to provide operatingpower to one or more components of the low-field MRI system 1600. Forexample, power management system 1610 may include one or more powersupplies, gradient power amplifiers, transmit coil amplifiers, and/orany other suitable power electronics needed to provide suitableoperating power to energize and operate components of the low-field MRIsystem 1600.

As illustrated in FIG. 16, power management system 1610 comprises powersupply 1612, amplifier(s) 1614, transmit/receive switch 1616, andthermal management components 1618. Power supply 1612 includeselectronics to provide operating power to magnetic components 1620 ofthe low-field MRI system 1600. For example, in some embodiments, powersupply 1612 may include electronics to provide operating power to one ormore B₀ coils (e.g., B₀ magnet 1622) to produce the main magnetic fieldfor the low-field MRI system, one or more shim coils 1624, and/or one ormore gradient coils 1628. In some embodiments, power supply 1612 may bea unipolar, continuous wave (CW) power supply, however, any suitablepower supply may be used. Transmit/receive switch 1616 may be used toselect whether RF transmit coils or RF receive coils are being operated.

In some embodiments, amplifier(s) 1614 may include one or more RFreceive (Rx) pre-amplifiers that amplify MR signals detected by one ormore RF receive coils (e.g., coils 1624), one or more RF transmit (Tx)amplifiers configured to provide power to one or more RF transmit coils(e.g., coils 1626), one or more gradient power amplifiers configured toprovide power to one or more gradient coils (e.g., gradient coils 1628),and/or one or more shim amplifiers configured to provide power to one ormore shim coils (e.g., shim coils 1624).

In some embodiments, thermal management components 1618 provide coolingfor components of low-field MRI system 1600 and may be configured to doso by facilitating the transfer of thermal energy generated by one ormore components of the low-field MRI system 1600 away from thosecomponents. Thermal management components 1618 may include, withoutlimitation, components to perform water-based or air-based cooling,which may be integrated with or arranged in close proximity to MRIcomponents that generate heat including, but not limited to, B₀ coils,gradient coils, shim coils, and/or transmit/receive coils. Thermalmanagement components 1618 may include any suitable heat transfer mediumincluding, but not limited to, air and water, to transfer heat away fromcomponents of the low-field MRI system 1600.

As illustrated in FIG. 16, low-field MRI system 1600 includes controller1606 (also referred to as a console) having control electronics to sendinstructions to and receive information from power management system1610. Controller 1606 may be configured to implement one or more pulsesequences, which are used to determine the instructions sent to powermanagement system 1610 to operate the magnetic components 1620 in adesired sequence. For example, controller 1606 may be configured tocontrol the power management system 1610 to operate the magneticcomponents 1620 in accordance with a balanced steady-state freeprecession (bSSFP) pulse sequence, a low-field gradient echo pulsesequence, a low-field spin echo pulse sequence, a low-field inversionrecovery pulse sequence, arterial spin labeling, diffusion weightedimaging (DWI), and/or any other suitable pulse sequence. Controller 1606may be implemented as hardware, software, or any suitable combination ofhardware and software, as aspects of the disclosure provided herein arenot limited in this respect.

In some embodiments, controller 1606 may be configured to implement apulse sequence by obtaining information about the pulse sequence frompulse sequences repository 1608, which stores information for each ofone or more pulse sequences. Information stored by pulse sequencesrepository 1608 for a particular pulse sequence may be any suitableinformation that allows controller 1606 to implement the particularpulse sequence. For example, information stored in pulse sequencesrepository 1608 for a pulse sequence may include one or more parametersfor operating magnetics components 1620 in accordance with the pulsesequence (e.g., parameters for operating the RF transmit and receivecoils 1626, parameters for operating gradient coils 1628, etc.), one ormore parameters for operating power management system 1610 in accordancewith the pulse sequence, one or more programs comprising instructionsthat, when executed by controller 1606, cause controller 1606 to controlsystem 1600 to operate in accordance with the pulse sequence, and/or anyother suitable information. Information stored in pulse sequencesrepository 1608 may be stored on one or more non-transitory storagemedia.

As illustrated in FIG. 16, in some embodiments, controller 1606 mayinteract with computing device 1604 programmed to process received MRdata (which, in some embodiments, may be spatial frequency domain MRdata). For example, computing device 1604 may process received MR datato generate one or more MR images using any suitable imagereconstruction process(es) including using any of the techniquesdescribed herein that make use of neural network models to generate MRimages from spatial frequency MR data. For example, computing device1604 may perform any of the processes described herein with reference toFIGS. 2A, 2B, 2C, 2D, and 14. Controller 1606 may provide informationabout one or more pulse sequences to computing device 1604 for theprocessing of data by the computing device. For example, controller 1606may provide information about one or more pulse sequences to computingdevice 1604 and the computing device may perform an image reconstructionprocess based, at least in part, on the provided information.

In some embodiments, computing device 1604 may be any electronic deviceor devices configured to process acquired MR data and generate one ormore images of the subject being imaged. In some embodiments, computingdevice 1604 may include a fixed electronic device such as a desktopcomputer, a server, a rack-mounted computer, or any other suitable fixedelectronic device that may be configured to process MR data and generateone or more images of the subject being imaged. Alternatively, computingdevice 1604 may be a portable device such as a smart phone, a personaldigital assistant, a laptop computer, a tablet computer, or any otherportable device that may be configured to process MR data and generateone or images of the subject being imaged. In some embodiments,computing device 1304 may comprise multiple computing devices of anysuitable type, as the aspects of the technology described herein are notlimited in this respect.

In some embodiments, a user 1602 may interact with computing device 1604to control aspects of the low-field MR system 1600 (e.g., program thesystem 1600 to operate in accordance with a particular pulse sequence,adjust one or more parameters of the system 1600, etc.) and/or viewimages obtained by the low-field MR system 1600. According to someembodiments, computing device 1604 and controller 1606 form a singlecontroller, while in other embodiments, computing device 1604 andcontroller 1606 each comprise one or more controllers. It should beappreciated that the functionality performed by computing device 1604and controller 1606 may be distributed in any way over any combinationof one or more controllers, as the aspects of the technology describedherein are not limited for use with any particular implementation orarchitecture.

FIGS. 17A and 17B illustrate bi-planar permanent magnet configurationsfor a B₀ magnet, in accordance with some embodiments of the technologydescribed herein. FIG. 17A illustrates a permanent B₀ magnet 2100, inaccordance with some embodiments. In the illustrated embodiment, B₀magnet 2100 is formed by permanent magnets 2110 a and 2110 b arranged ina bi-planar geometry and a yoke 2120 that captures electromagnetic fluxproduced by the permanent magnets and transfers the flux to the opposingpermanent magnet to increase the flux density between permanent magnets2110 a and 2110 b. Each of permanent magnets 2110 a and 2110 b is formedfrom a plurality of concentric permanent magnet rings. In particular, asvisible in FIG. 17A, permanent magnet 2110 b comprises an outer ring ofpermanent magnets 2114 a, a middle ring of permanent magnets 2114 b, aninner ring of permanent magnets 2114 c, and a permanent magnet disk 2114d at the center. Though shown with four concentric permanent magnetrings, permanent magnet 2110 b (and permanent magnet 2110 a) may haveany suitable number of permanent magnet rings, as aspects of thetechnology described herein are not limited in this respect. Permanentmagnet 2110 a may be formed substantially identically to permanentmagnet 2110 b and, for example, comprise the same set of permanentmagnet rings as permanent magnet 2110 b.

The permanent magnet material used may be selected depending on thedesign requirements of the system. For example, according to someembodiments, the permanent magnets (or some portion thereof) may be madeof NdFeB, which produces a magnetic field with a relatively highmagnetic field per unit volume of material once magnetized. In someembodiments, SmCo material is used to form the permanent magnets, orsome portion thereof. While NdFeB produces higher field strengths (andin general is less expensive than SmCo), SmCo exhibits less thermaldrift and thus provides a more stable magnetic field in the face oftemperature fluctuations. Other types of permanent magnet material(s)may be used as well, as the aspects of the technology described hereinare not limited in this respect. In general, the type or types ofpermanent magnet material utilized will depend, at least in part, on thefield strength, temperature stability, weight, cost and/or ease of userequirements of a given B₀ magnet implementation.

In some embodiments, the permanent magnet rings are sized and arrangedto produce a homogenous field of a desired strength in the imagingregion (field of view) between permanent magnets 2110 a and 2110 b. Inthe exemplary embodiment illustrated in FIG. 17A, each permanent magnetring comprises a plurality segments, each segment formed using aplurality of permanent magnet blocks stacked in the radial direction andpositioned adjacent to one another about the periphery to form therespective ring. The inventors have appreciated that by varying thewidth (in the direction tangent to the ring) of each permanent magnet,less waste of useful space may be achieved while using less material.For example, the space between stacks that does not produce usefulmagnetic fields can be reduced by varying the width of the blocks, forexample, as function of the radial position of the block, allowing for acloser fit to reduce wasted space and maximize the amount of magneticfield that can be generated in a given space. The dimensions of theblocks may also be varied in any desired way to facilitate theproduction of a magnetic field of desired strength and homogeneity. Forexample, in some embodiments, the heights of the blocks different ringsmay be different from one another and/or the heights of one or moreblocks within a particular ring may be different from one another inorder to achieve a magnetic field of desired strength and homogeneity.

As shown in FIG. 17A, B₀ magnet 2100 further comprises yoke 2120configured and arranged to capture magnetic flux generated by permanentmagnets 2110 a and 2110 b and direct it to the opposing side of the B₀magnet to increase the flux density in between permanent magnets 2110 aand 2110 b, increasing the field strength within the field of view ofthe B₀ magnet. By capturing magnetic flux and directing it to the regionbetween permanent magnets 2110 a and 2110 b, less permanent magnetmaterial can be used to achieve a desired field strength, thus reducingthe size, weight and cost of the B₀ magnet 2100. Alternatively, forgiven permanent magnets, the field strength can be increased, thusimproving the SNR of the system without having to use increased amountsof permanent magnet material. For exemplary B₀ magnet 2100, yoke 2120comprises a frame 2122 and plates 2124 a and 2124 b. Plates 2124 a and2124 b may capture magnetic flux generated by permanent magnets 2110 aand 2110 b and direct it to frame 2122 to be circulated via the magneticreturn path of the yoke to increase the flux density in the field ofview of the B₀ magnet. Yoke 2120 may be constructed of any desiredferromagnetic material, for example, low carbon steel, CoFe and/orsilicon steel, etc. to provide the desired magnetic properties for theyoke. In some embodiments, plates 2124 a and 2124 b (and/or frame 2122or portions thereof) may be constructed of silicon steel or the like inareas where the gradient coils could most prevalently induce eddycurrents.

Exemplary frame 2122 comprises arms 2123 a and 2123 b that attach toplates 2124 a and 2124 b, respectively, and supports 2125 a and 2125 bproviding the magnetic return path for the flux generated by thepermanent magnets. The arms are generally designed to reduce the amountof material needed to support the permanent magnets while providingsufficient cross-section for the return path for the magnetic fluxgenerated by the permanent magnets. Frame 2122 has two supports within amagnetic return path for the B₀ field produced by the B₀ magnet.Supports 2125 a and 2125 b are produced with a gap 2127 formed between,providing a measure of stability to the frame and/or lightness to thestructure while providing sufficient cross-section for the magnetic fluxgenerated by the permanent magnets. For example, the cross-sectionneeded for the return path of the magnetic flux can be divided betweenthe two support structures, thus providing a sufficient return pathwhile increasing the structural integrity of the frame.

FIG. 17B illustrates a B₀ magnet 2200, in accordance with someembodiments. B₀ magnet 2200 may share design components with B₀ magnet2100 illustrated in FIG. 17A. In particular, B₀ magnet 2200 is formed bypermanent magnets 2210 a and 2210 b arranged in a bi-planar geometrywith a yoke 2220 coupled thereto to capture electromagnetic fluxproduced by the permanent magnets and transfer the flux to the opposingpermanent magnet to increase the flux density between permanent magnets2210 a and 2210 b. Each of permanent magnets 2210 a and 2210 b is formedfrom a plurality of concentric permanent magnets, as shown by permanentmagnet 2210 b comprising an outer ring of permanent magnets 2214 a, amiddle ring of permanent magnets 2214 b, an inner ring of permanentmagnets 2214 c, and a permanent magnet disk 2214 d at the center.Permanent magnet 2210 a may comprise the same set of permanent magnetelements as permanent magnet 2210 b. The permanent magnet material usedmay be selected depending on the design requirements of the system(e.g., NdFeB, SmCo, etc. depending on the properties desired).

The permanent magnet rings are sized and arranged to produce ahomogenous field of a desired strength in the central region (field ofview) between permanent magnets 2210 a and 2210 b. In the exemplaryembodiment of FIG. 17B, each permanent magnet ring comprises a pluralityof circular arc segments sized and positioned to produce a desired B₀magnetic field. In a similar manner to yoke 2120 illustrated in FIG.17A, yoke 2220 is configured and arranged to capture magnetic fluxgenerated by permanent magnets 2210 a and 2210 b and direct it to theopposing side of the B₀ magnet to increase the flux density betweenpermanent magnets 2210 a and 2210 b. Yoke 2220 thereby increases thefield strength within the field of view of the B₀ magnet with lesspermanent magnet material, reducing the size, weight and cost of the B₀magnet. Yoke 2220 also comprises a frame 2222 and plates 2224 a and 2224b that, in a manner similar to that described above in connection withyoke 2220, captures and circulates magnetic flux generated by thepermanent magnets 2210 a and via the magnetic return path of the yoke toincrease the flux density in the field of view of the B₀ magnet. Thestructure of yoke 2220 may be similar to that described above to providesufficient material to accommodate the magnetic flux generated by thepermanent magnets and providing sufficient stability, while minimizingthe amount of material used to, for example, reduce the cost and weightof the B₀ magnet.

Because a permanent B₀ magnet, once magnetized, will produce its ownpersistent magnetic field, power is not required to operate thepermanent B₀ magnet to generate its magnetic field. As a result, asignificant (often dominant) contributor to the overall powerconsumption of an MRI system is eliminated through the use of apermanent magnet (as opposed to, e.g., an electro-magnet which requirespower), facilitating the development of an MRI system that can bepowered using mains electricity (e.g., via a standard wall outlet orcommon large household appliance outlets). As described above, theinventors have developed low power, portable low-field MRI systems thatcan be deployed in virtually any environment and that can be brought tothe patient who will undergo an imaging procedure. In this way, patientsin emergency rooms, intensive care units, operating rooms and a host ofother locations can benefit from MRI in circumstances where MRI isconventionally unavailable.

FIGS. 18A and 18B illustrate views of a portable MRI system 3800, inaccordance with some embodiments of the technology described herein.Portable MRI system 3800 comprises a B₀ magnet 3810 formed in part by anupper magnet 3810 a and a lower magnet 3810 b having a yoke 3820 coupledthereto to increase the flux density within the imaging region. The B₀magnet 3810 may be housed in magnet housing 3812 along with gradientcoils 3815 (e.g., any of the gradient coils described in U.S.application Ser. No. 14/845,652, titled “Low Field Magnetic ResonanceImaging Methods and Apparatus” and filed on Sep. 4, 2015, which isherein incorporated by reference in its entirety). In some embodiments,B₀ magnet 3810 comprises an electromagnet. In some embodiments, B₀magnet 3810 comprises a permanent magnet (e.g., any permanent magnetdescribed in U.S. application Ser. No. 15/640,369, titled “LOW-FIELDMAGNETIC RESONANCE IMAGING METHODS AND APPARATUS,” filed on Jun. 30,2017, which is incorporated by reference herein in its entirety). Forexample, in some embodiments, B₀ magnet 3810 may be the permanent magnet2100 described with reference to FIG. 17A or the permanent magnet 2200described with reference to FIG. 17B.

Illustrative portable MRI system 3800 further comprises a base 3850housing the electronics that operates the MRI system. For example, base3850 may house electronics including, but not limited to, one or moregradient power amplifiers, an on-system computer, a power distributionunit, one or more power supplies, and/or any other power componentsconfigured to operate the MRI system using mains electricity (e.g., viaa connection to a standard wall outlet and/or a large appliance outlet).For example, base 3870 may house low power components, such as thosedescribed herein, enabling at least in part the portable MRI system tobe powered from readily available wall outlets. Accordingly, portableMRI system 3800 can be brought to the patient and plugged into a walloutlet in his or her vicinity.

Portable MRI system 3800 further comprises moveable slides 3860 that canbe opened and closed and positioned in a variety of configurations.Slides 3860 include electromagnetic shielding 3865, which can be madefrom any suitable conductive or magnetic material, to form a moveableshield to attenuate electromagnetic noise in the operating environmentof the portable MRI system to shield the imaging region from at leastsome electromagnetic noise. As used herein, the term electromagneticshielding refers to conductive or magnetic material configured toattenuate the electromagnetic field in a spectrum of interest andpositioned or arranged to shield a space, object and/or component ofinterest. In the context of an MRI system, electromagnetic shielding maybe used to shield electronic components (e.g., power components, cables,etc.) of the MRI system, to shield the imaging region (e.g., the fieldof view) of the MRI system, or both.

The degree of attenuation achieved from electromagnetic shieldingdepends on a number of factors including the type material used, thematerial thickness, the frequency spectrum for which electromagneticshielding is desired or required, the size and shape of apertures in theelectromagnetic shielding (e.g., the size of the spaces in a conductivemesh, the size of unshielded portions or gaps in the shielding, etc.)and/or the orientation of apertures relative to an incidentelectromagnetic field. Thus, electromagnetic shielding refers generallyto any conductive or magnetic barrier that acts to attenuate at leastsome electromagnetic radiation and that is positioned to at leastpartially shield a given space, object or component by attenuating theat least some electromagnetic radiation.

It should be appreciated that the frequency spectrum for which shielding(attenuation of an electromagnetic field) is desired may differdepending on what is being shielded. For example, electromagneticshielding for certain electronic components may be configured toattenuate different frequencies than electromagnetic shielding for theimaging region of the MRI system. Regarding the imaging region, thespectrum of interest includes frequencies which influence, impact and/ordegrade the ability of the MRI system to excite and detect an MRresponse. In general, the spectrum of interest for the imaging region ofan MRI system correspond to the frequencies about the nominal operatingfrequency (i.e., the Larmor frequency) at a given B₀ magnetic fieldstrength for which the receive system is configured to or capable ofdetecting. This spectrum is referred to herein as the operating spectrumfor the MRI system. Thus, electromagnetic shielding that providesshielding for the operating spectrum refers to conductive or magneticmaterial arranged or positioned to attenuate frequencies at least withinthe operating spectrum for at least a portion of an imaging region ofthe MRI system.

In portable MRI system 3800 illustrated in FIGS. 18A and 18B, themoveable shields are thus configurable to provide shielding in differentarrangements, which can be adjusted as needed to accommodate a patient,provide access to a patient, and/or in accordance with a given imagingprotocol. For example, for an imaging procedure such as a brain scan,once the patient has been positioned, slides 3960 can be closed, forexample, using handle 3862 to provide electromagnetic shielding 3965around the imaging region except for the opening that accommodates thepatient's upper torso. As another example, for an imaging procedure suchas a knee scan, slides 3960 may be arranged to have openings on bothsides to accommodate the patient's leg or legs. Accordingly, moveableshields allow the shielding to be configured in arrangements suitablefor the imaging procedure and to facilitate positioning the patientappropriately within the imaging region.

In some embodiments, a noise reduction system comprising one or morenoise reduction and/or compensation techniques may be performed tosuppress at least some of the electromagnetic noise that is not blockedor sufficiently attenuated by shielding 3865. In particular, theinventors have developed noise reduction systems configured to suppress,avoid and/or reject electromagnetic noise in the operating environmentin which the MRI system is located. According to some embodiments, thesenoise suppression techniques work in conjunction with the moveableshields to facilitate operation in the various shielding configurationsin which the slides may be arranged. For example, when slides 3960 areopened, increased levels of electromagnetic noise will likely enter theimaging region via the openings. As a result, the noise suppressioncomponent will detect increased electromagnetic noise levels and adaptthe noise suppression and/or avoidance response accordingly. Due to thedynamic nature of the noise suppression and/or avoidance techniquesdescribed herein, the noise reduction system is configured to beresponsive to changing noise conditions, including those resulting fromdifferent arrangements of the moveable shields. Thus, a noise reductionsystem in accordance with some embodiments may be configured to operatein concert with the moveable shields to suppress electromagnetic noisein the operating environment of the MRI system in any of the shieldingconfigurations that may be utilized, including configurations that aresubstantially without shielding (e.g., configurations without moveableshields).

To ensure that the moveable shields provide shielding regardless of thearrangements in which the slides are placed, electrical gaskets may bearranged to provide continuous shielding along the periphery of themoveable shield. For example, as shown in FIG. 18B, electrical gaskets3867 a and 3867 b may be provided at the interface between slides 3860and magnet housing to maintain to provide continuous shielding alongthis interface. According to some embodiments, the electrical gasketsare beryllium fingers or beryllium-copper fingers, or the like (e.g.,aluminum gaskets), that maintain electrical connection between shields3865 and ground during and after slides 3860 are moved to desiredpositions about the imaging region.

To facilitate transportation, a motorized component 3880 is provide toallow portable MRI system to be driven from location to location, forexample, using a control such as a joystick or other control mechanismprovided on or remote from the MRI system. In this manner, portable MRIsystem 3800 can be transported to the patient and maneuvered to thebedside to perform imaging.

The portable MRI systems described herein may be operated from aportable electronic device, such as a notepad, tablet, smartphone, etc.For example, tablet computer 3875 may be used to operate portable MRIsystem to run desired imaging protocols and to view the resultingimages. Tablet computer 3875 may be connected to a secure cloud totransfer images for data sharing, telemedicine, and/or deep learning onthe data sets. Any of the techniques of utilizing network connectivitydescribed in U.S. application Ser. No. 14/846,158, titled “AutomaticConfiguration of a Low Field Magnetic Resonance Imaging System,” filedSep. 4, 2015, which is herein incorporated by reference in its entirety,may be utilized in connection with the portable MRI systems describedherein.

As discussed above, FIG. 18C illustrates a portable MRI system 3900 thathas been transported to a patient's bedside to perform a brain scan.FIG. 18D illustrates portable MRI system 3900 that has been transportedto a patient's bedside to perform a scan of the patient's knee. As shownin FIG. 18D, shield 3960 have electrical gaskets 3867 c.

It should be appreciated that the electromagnetic shields illustrated inFIGS. 18A-18D are exemplary and providing shielding for an MRI system isnot limited to the example electromagnetic shielding described herein.Electromagnetic shielding can be implemented in any suitable way usingany suitable materials. For example, electromagnetic shielding may beformed using conductive meshes, fabrics, etc. that can provide amoveable “curtain” to shield the imaging region. Electromagneticshielding may be formed using one or more conductive straps (e.g., oneor more strips of conducting material) coupled to the MRI system aseither a fixed, moveable or configurable component to shield the imagingregion from electromagnetic interference, some examples of which aredescribed in further detail below. Electromagnetic shielding may beprovided by embedding materials in doors, slides, or any moveable orfixed portion of the housing. Electromagnetic shields may be deployed asfixed or moveable components, as the aspects are not limited in thisrespect.

FIG. 19 is a diagram of an illustrative computer system on whichembodiments described herein may be implemented. An illustrativeimplementation of a computer system 1900 that may be used in connectionwith any of the embodiments of the disclosure provided herein is shownin FIG. 19. For example, the processes described with reference to FIGS.2A-2D and 14 may be implemented on and/or using computer system 1900. Asanother example, the computer system 1900 may be used to train and/oruse any of the neural network statistical models described herein. Thecomputer system 1900 may include one or more processors 1910 and one ormore articles of manufacture that comprise non-transitorycomputer-readable storage media (e.g., memory 1920 and one or morenon-volatile storage media 1930). The processor 1910 may control writingdata to and reading data from the memory 1920 and the non-volatilestorage device 1930 in any suitable manner, as the aspects of thedisclosure provided herein are not limited in this respect. To performany of the functionality described herein, the processor 1910 mayexecute one or more processor-executable instructions stored in one ormore non-transitory computer-readable storage media (e.g., the memory1920), which may serve as non-transitory computer-readable storage mediastoring processor-executable instructions for execution by the processor1910.

Having thus described several aspects and embodiments of the technologyset forth in the disclosure, it is to be appreciated that variousalterations, modifications, and improvements will readily occur to thoseskilled in the art. Such alterations, modifications, and improvementsare intended to be within the spirit and scope of the technologydescribed herein. For example, those of ordinary skill in the art willreadily envision a variety of other means and/or structures forperforming the function and/or obtaining the results and/or one or moreof the advantages described herein, and each of such variations and/ormodifications is deemed to be within the scope of the embodimentsdescribed herein. Those skilled in the art will recognize, or be able toascertain using no more than routine experimentation, many equivalentsto the specific embodiments described herein. It is, therefore, to beunderstood that the foregoing embodiments are presented by way ofexample only and that, within the scope of the appended claims andequivalents thereto, inventive embodiments may be practiced otherwisethan as specifically described. In addition, any combination of two ormore features, systems, articles, materials, kits, and/or methodsdescribed herein, if such features, systems, articles, materials, kits,and/or methods are not mutually inconsistent, is included within thescope of the present disclosure.

The above-described embodiments can be implemented in any of numerousways. One or more aspects and embodiments of the present disclosureinvolving the performance of processes or methods may utilize programinstructions executable by a device (e.g., a computer, a processor, orother device) to perform, or control performance of, the processes ormethods. In this respect, various inventive concepts may be embodied asa computer readable storage medium (or multiple computer readablestorage media) (e.g., a computer memory, one or more floppy discs,compact discs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other tangible computer storage medium) encoded with one ormore programs that, when executed on one or more computers or otherprocessors, perform methods that implement one or more of the variousembodiments described above. The computer readable medium or media canbe transportable, such that the program or programs stored thereon canbe loaded onto one or more different computers or other processors toimplement various ones of the aspects described above. In someembodiments, computer readable media may be non-transitory media.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects as described above. Additionally,it should be appreciated that according to one aspect, one or morecomputer programs that when executed perform methods of the presentdisclosure need not reside on a single computer or processor, but may bedistributed in a modular fashion among a number of different computersor processors to implement various aspects of the present disclosure.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconvey relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

When implemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer, as non-limitingexamples. Additionally, a computer may be embedded in a device notgenerally regarded as a computer but with suitable processingcapabilities, including a Personal Digital Assistant (PDA), a smartphoneor any other suitable portable or fixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audibleformats.

Such computers may be interconnected by one or more networks in anysuitable form, including a local area network or a wide area network,such as an enterprise network, and intelligent network (IN) or theInternet. Such networks may be based on any suitable technology and mayoperate according to any suitable protocol and may include wirelessnetworks, wired networks or fiber optic networks.

Also, as described, some aspects may be embodied as one or more methods.The acts performed as part of the method may be ordered in any suitableway. Accordingly, embodiments may be constructed in which acts areperformed in an order different than illustrated, which may includeperforming some acts simultaneously, even though shown as sequentialacts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” “composed of,” and the like are tobe understood to be open-ended, i.e., to mean including but not limitedto. Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively.

The terms “approximately” and “about” may be used to mean within ±20% ofa target value in some embodiments, within ±10% of a target value insome embodiments, within ±5% of a target value in some embodiments,within ±2% of a target value in some embodiments. The terms“approximately” and “about” may include the target value.

What is claimed is:
 1. A method, comprising: generating a magneticresonance (MR) image from input MR spatial frequency data using a neuralnetwork model that comprises: a first neural network sub-modelconfigured to process spatial frequency domain data; and a second neuralnetwork sub-model configured to process image domain data.
 2. The methodof claim 1, wherein the generating comprises: processing the input MRspatial frequency data using the first neural network sub-model toobtain output MR spatial frequency data; transforming the output MRspatial frequency data to the image domain to obtain input image-domaindata; and processing the input image-domain data using the second neuralnetwork sub-model to obtain the MR image.
 3. The method of claim 1,wherein the first neural network sub-model includes at least oneconvolutional layer and at least one transposed convolutional layer. 4.The method of claim 1, wherein the first neural network sub-modelincludes at least one locally-connected layer.
 5. The method of claim 1,wherein the first neural network sub-model includes at least one dataconsistency layer.
 6. The method of claim 1, wherein the first neuralnetwork sub-model comprises a data consistency block implemented atleast in part using a non-uniform fast Fourier transform and the secondneural network sub-model comprises a convolutional neural network block.7. The method of claim 2, wherein the first neural network sub-modelincludes at least one convolutional layer, a locally-connected layer,and at least one transposed convolutional layer, and wherein processingthe input MR spatial frequency data using the first neural networksub-model comprises: applying the at least one convolutional layer tothe input MR spatial frequency data; applying the locally-connectedlayer to data obtained using output of the at least one convolutionallayer; and applying the at least one transposed convolutional layer todata obtained using output of the locally-connected layer.
 8. The methodof claim 7, wherein the first neural network sub-model includes acomplex-conjugate symmetry layer, and wherein processing the input MRspatial frequency data using the first neural network sub-modelcomprises: applying the complex-conjugate symmetry layer to dataobtained using output of the at least one transposed convolutionallayer.
 9. The method of claim 7, wherein the first neural networksub-model includes a data consistency layer, and wherein processing theinput MR spatial frequency data using the first neural network sub-modelcomprises: applying the data consistency layer to data obtained usingoutput of the complex-conjugate symmetry layer.
 10. The method of claim1, wherein the first neural network sub-model includes at least onefully-connected layer.
 11. The method of claim 1, wherein the firstneural network sub-model includes a fully-connected layer, the methodfurther comprising: applying the fully-connected layer to a real part ofthe spatial frequency domain data; and applying the fully-connectedlayer to an imaginary part of the spatial frequency domain data.
 12. Themethod of claim 1, wherein the first neural network sub-model includes afirst fully-connected and a second fully connected layer, the methodfurther comprising: applying the first fully-connected layer to a realpart of the spatial frequency domain data; applying the secondfully-connected layer to an imaginary part of the spatial frequencydomain data.
 13. The method of claim 12, wherein the first and secondfully-connected layers share at least some weights.
 14. The method ofclaim 12, further comprising: transforming output of the first andsecond fully-connected layers using a Fourier transformation to obtainimage-domain data; and providing the image-domain data as input to thesecond neural network sub-model.
 15. The method of claim 1, wherein thesecond neural network sub-model comprises a series of blocks comprisingrespective sets of neural network layers, each of the plurality ofblocks comprising at least one convolutional layer and at least onetransposed convolutional layer.
 16. The method of claim 15, wherein eachof the plurality of blocks further comprises: a Fourier transformationlayer, a data consistency layer, and an inverse Fourier transformationlayer.
 17. The method of claim 1, further comprising: training theneural network model using a set of high-field images to obtain a firsttrained neural network model; and adapting the first neural networkmodel by using a set of low-field images.
 18. The method of claim 1,wherein the spatial frequency domain data is under-sampled relative to aNyquist criterion.
 19. A system, comprising: at least one computerhardware processor; and at least one non-transitory computer-readablestorage medium storing processor-executable instructions that, whenexecuted by the at least one computer hardware processor, cause the atleast one computer hardware processor to perform: generating a magneticresonance (MR) image from MR spatial frequency data using a neuralnetwork model that comprises: a first neural network portion configuredto process data in a spatial frequency domain; and a second neuralnetwork portion configured to process data in an image domain.
 20. Amagnetic resonance imaging (MRI) system, comprising: a magnetics systemcomprising: a B₀ magnet configured to provide a B₀ field for the MRIsystem; gradient coils configured to provide gradient fields for the MRIsystem; and at least one RF coil configured to detect magnetic resonance(MR) signals; a controller configured to: control the magnetics systemto acquire MR spatial frequency data; generate an MR image from MRspatial frequency data using a neural network model that comprises: afirst neural network portion configured to process data in a spatialfrequency domain; and a second neural network portion configured toprocess data in an image domain.